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RQ-21A  Blackjack  Small  Tactical  Unmanned  Aircraft  System 


Executive  Summary 


This  document  reports  on  the  evaluation  of  test  adequacy,  operational  effectiveness,  and 
operational  suitability  of  the  RQ-21A  Blackjack  Small  Tactical  Unmanned  Aircraft  System 
(STUAS)  during  Initial  Operational  Test  and  Evaluation  (IOT&E).  The  Navy’s  Commander, 
Operational  Test  and  Evaluation  Force  (COMOPTEVFOR),  with  assistance  from  the  Marine 
Corps  Operational  Test  and  Evaluation  Activity  (MCOTEA),  conducted  the  RQ-21A  IOT&E 
between  January  and  December  2014.  The  Navy  and  Marine  Corps  plan  to  use  the  results  of 
IOT&E  to  inform  the  Services’  Full-Rate  Production  (FRP)  decision,  which  was  originally 
planned  for  October  2015,  and  is  now  scheduled  to  occur  in  September  2016.  Additional  testing 
is  planned  to  occur  before  the  September  2016  FRP.  DOT&E  intends  to  provide  an  updated 
operational  assessment  prior  to  the  FRP  to  further  support  that  decision.  The  RQ-21 A  is  an 
Acquisition  Category  II  program. 

The  RQ-21  A  is  not  operationally  effective.  The  detachment  equipped  with  RQ-21  A  is 
not  effective  in  supporting  the  ground  commander’s  mission  because  of  an  inability  to  have  an 
unmanned  aircraft  arrive  on  station  at  the  designated  time  and  remain  on  station  for  the  duration 
of  the  tasked  period.  The  electro-optical/infrared  sensor  provides  accurate  target  locations. 
Operators  using  the  sensors  were  not  able  to  correctly  classify  1 -meter  targets,  which  is 
necessary  for  consistent  classification  of  hostile/non-hostile  intent  on  the  part  of  individuals  by 
classifying  such  items  as  rifles,  rocket-propelled  grenade  launchers,  and  shovels. 

The  RQ-21  A  is  not  operationally  suitable.  The  RQ-21  A  demonstrated  a  Mean  Flight 
Hour  between  Abort  for  the  System  (MFHBAsys)  of  15.2  hours  versus  the  50-hour  requirement. 
Because  of  air  vehicle  reliability,  overall  system  availability  did  not  meet  the  80  percent  key 
performance  parameter  threshold  (demonstrated  value  =  66.9  percent).  The  average  service  life 
of  the  propulsion  modules  was  48.9  hours,  which  does  not  meet  the  manufacturer’s  stated 
100-hour  capability.  Production  quality  control  issues  contributed  to  the  system’s  poor 
reliability  and  availability. 

Cybersecurity  testing  demonstrated  that  the  system  has  exploitable  vulnerabilities.  The 
classified  appendix  contains  additional  detail  regarding  cybersecurity  testing. 

The  RQ-21  A  testing  was  adequate  and  executed  in  accordance  with  the  DOT&E- 
approved  test  plan. 

System  Description  and  Mission 

The  RQ-21  A  system  includes  five  air  vehicles  (each  air  vehicle  is  equipped  with  an 
electro-optical/inffared  payload  and  a  communications  relay  payload),  two  ground  control 
stations  (each  ground  control  station  has  two  operator  workstations),  four  ground  data  terminals, 
a  launcher,  a  recovery  system,  and  government-furnished  vehicles  and  support  systems. 

Marine  Corps  commanders  intend  to  use  the  RQ-21  A  to  provide  units  ashore  with  an 
organic  battlefield  intelligence,  surveillance,  and  reconnaissance  capability  to  reduce  their 
dependence  on  higher  headquarters  for  such  assets.  In  addition  to  operating  from  land  bases, 
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detachments  from  Marine  Corps  unmanned  aerial  vehicle  squadrons  will  embark  personnel  and 
equipment  aboard  L-class  ships  to  support  operations  in  the  maritime  domain. 

Test  Adequacy 

Operational  testing  of  the  RQ-21A  was  adequate  to  support  an  evaluation  of  the 
operational  effectiveness  and  operational  suitability  of  the  system.  The  Navy  and  Marine  Corps 
conducted  testing  from  January  to  December  2014  in  three  phases  from  three  different  locations. 
The  test  locations  in  sequence  were  Marine  Corps  Air-Ground  Combat  Center  Twentynine 
Palms,  California;  Marine  Corps  Base  Camp  Lejeune,  North  Carolina;  and  aboard  USS 
Anchorage  (LPD  23).  Total  flight  time  accrued  from  all  three  test  sites  was  233.6  hours  during 
38  flights. 

In  May  2014,  the  Marine  Corps  deployed  an  Early  Operational  Capability  RQ-21A 
system  to  support  overseas  combat  operations.  This  deployed  RQ-21A  detachment  consisted  of 
up  to  eight  air  vehicles,  two  launchers,  and  two  recovery  systems.  Between  May  and  September, 
the  system  flew  approximately  121  flights  for  995  flight  hours.  Each  day  the  detachment  sent  a 
daily  summary  containing  the  status  of  each  major  piece  of  equipment  and  a  narrative  summary 
of  the  day’s  flight  activities.  Data  collected  by  the  Early  Operational  Capability  detachment 
while  deployed  overseas  provided  additional  insights  into  system  reliability. 

Previous  Operational  Testing  Results  Revisited  in  IOT&E 

In  support  of  the  RQ-21A  Milestone  C,  the  Navy  conducted  Operational  Test  B-2 
(OT-B2)  from  September  to  November  2012,  from  which  DOT&E  provided  an  assessment  of 
demonstrated  performance.  Subsequent  to  that  test,  the  Navy  addressed  four  of  the  seven 
recommendations  contained  in  the  DOT&E  April  2013  memorandum  (updated  the  reliability 
growth  curve,  verified  fuselage  structural  integrity,  identified  and  corrected  the  root  causes  for 
the  excessive  number  of  payload  resets,  quantified  and  compared  the  power  requirements  for  the 
system’s  ground  components  to  the  generating  capabilities  of  standard  Marine  Corps  generators). 
The  Navy  did  not  fully  address  two  recommendations  from  this  memo,  and  both  issues 
(strengthening  quality  control  processes  and  communications  relay  payload  frequency  changes) 
adversely  affected  system  performance  during  IOT&E.  DOT&E’s  final  recommendation  to 
provide  a  reliability  rationale  in  the  Capability  Production  Document  was  not  addressed  and 
would  have  provided  operational  insight  into  the  poor  system  reliability  demonstrated  during 
IOT&E. 

Based  upon  data  collected  during  OT-B2  in  2012,  DOT&E  determined  that  the  electro- 
optical/  infrared  payload  demonstrated  the  potential  to  provide  tactical  commanders  with 
accurate  target  acquisition  intelligence.  In  this  report,  DOT&E  determined  that,  while  the 
electro-optical/  infrared  sensor  provides  accurate  target  locations,  the  low  probability  of  correct 
classification  for  1 -meter  targets,  such  as  rifles,  shovels,  or  rocket-propelled  grenade  launchers, 
does  not  allow  for  consistent  determination  of  hostile/non-hostile  intent  on  the  part  of 
individuals.  The  difference  between  the  DOT&E  OT-B2  and  IOT&E  assessment  is  attributable 
to  changes  between  the  Capability  Development  Document  and  the  Capability  Production 
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Document  (CPD).  The  more  stringent  requirements  in  the  CPD  and  the  operating  environment 
of  IOT&E  resulted  in  poor  payload  performance. 

DOT&E  first  identified  the  communications  relay  payload  lack  of  in-flight  frequency 
change  capability  as  a  potential  deficiency  during  OT-B1  (January  2010)  and  highlighted  this 
again  in  the  DOT&E  2013  OT-B2  memorandum.  In  this  report,  DOT&E  determines  that  the 
communications  relay  payload  limits  the  commanders’  tactical  flexibility. 

Just  prior  to  OT-B2,  developmental  testing  experienced  a  fuselage  failure  that  resulted  in 
a  cracked  airframe,  possibly  attributable  to  production  quality  control  issues.  DOT&E 
recommended  that  the  Navy  strengthen  the  quality  control  process  in  those  areas  where 
manufacturing  shortcomings  could  result  in  air  vehicle  losses  in  order  to  minimize  the  potential 
for  future  mishaps.  During  IOT&E,  production  quality  control  issues  contributed  to  the  system’s 
poor  reliability  and  availability. 

During  OT-B2,  the  system  demonstrated  an  MFHBASys  of  10.6  hours,  below  the 
threshold  requirement  of  50  flight  hours.  The  demonstrated  MFHBAsys  of  15.2  hours  during 
IOT&E  continues  to  be  well  below  the  threshold  requirement.  This  contributed  towards  DOT&E 
finding  the  RQ-21A  not  suitable. 

Operational  Effectiveness 

The  RQ-21 A  is  not  operationally  effective.  Mission  accomplishment  for  the  detachment 
equipped  with  the  RQ-21  A  begins  with  the  unmanned  air  vehicle  arriving  at  the  desired  location, 
at  the  prescribed  time,  and  establishing  contact  with  the  supported  unit.  The  unmanned  aircraft 
should  then  be  capable  of  performing  assigned  tasks  for  the  duration  of  the  required  on-station 
time. 

Once  on  station,  the  air  vehicles  sensors  should  support  the  commander’s  intelligence 
requirements  by  providing  accurate  locations  and  reports  of  enemy  actions.  The  unmanned 
aircraft  should  support  additional  missions  for  which  it  is  equipped,  such  as  communications 
relay,  and  the  unmanned  aircraft  should  have  the  capability  of  accepting  dynamic  retasking  to 
support  the  commander’s  emergent  needs.  The  mission  concludes  with  the  safe  recovery  of  the 
air  vehicle  without  encountering  a  significant  amount  of  damage. 

The  Mission  Coverage  metric  measures  the  percentage  of  time  an  air  vehicle  is  on  station 
when  tasked  to  do  so.  This  metric  provides  an  indication  of  the  level  of  support  a  ground 
commander  can  expect  from  the  RQ-21  A.  During  the  IOT&E,  the  RQ-21  A-equipped  unit 
provided  coverage  during  68  percent  of  the  tasked  on-station  hours  (83.8  of  122.7  hours).  In 
essence,  RQ-21  A  was  absent  or  not  capable  of  providing  requisite  support  one-third  of  the  time 
the  commander  expected  to  have  support. 

The  RQ-21  A  demonstrated  a  low  probability  of  successfully  completing  realistic 
missions  during  IOT&E.  Nineteen  of  45  attempted  flights  (42  percent)  launched  on  time, 
supported  assigned  tasking,  and  remained  on  station  for  the  duration  of  the  assigned  period.  Of 
the  45  attempted  launches,  35  (78  percent)  provided  some  level  of  utility  to  ground  commanders 
(were  able  to  launch  and  provide  some  on-station  time). 


The  RQ-21 A  sensor  does  not  meet  one  of  the  two  target  classification  key  performance 
parameters  established  in  the  CPD.  The  electro-optical  sensor  is  required  to  provide  a  50  percent 
probability  of  correct  classification  for  1 -meter  linear  targets  (weapons  or  tools)  while  the 
infrared  sensor  is  required  to  support  50  percent  correct  classification  for  3-meter  sized  linear 
targets  (vehicle  chassis  type).  The  infrared  sensor  demonstrated  100  percent  correct 
classification  of  3 -meter  objects,  exceeding  the  key  performance  parameter  threshold.  For  the 
electro-optical  sensor,  regression  models  based  upon  IOT&E  data  indicate  a  probability  of 
correct  classification  of  41  percent  in  a  desert  environment  and  43  percent  in  the  maritime 
environment. 

The  communications  relay  payload  limits  the  commanders’  tactical  flexibility  and 
mission  accomplishment.  The  communications  relay  payload  is  constrained  to  a  single 
frequency  in  each  of  the  two  radios.  Before  launch,  maintainers  set  the  frequency  in  each  radio. 
Once  airborne,  the  only  interface  between  the  air  vehicle  and  the  communications  relay  payload 
is  a  power  connector.  Operators  can  power  the  communications  relay  payload  on  and  off,  but 
cannot  change  frequencies.  This  limits  the  communications  relay  payload  to  supporting 
preplanned  relay  missions  on  a  single  pair  of  pre-set  frequencies.  On  four  separate  occasions 
during  IOT&E,  the  RQ-21  A-equipped  unit  was  not  able  to  provide  support  to  ground  units 
because  the  communications  relay  payload  was  set  to  frequencies  different  from  those  in  use  by 
the  ground  units.  When  able  to  establish  compatible  frequencies,  the  payload  is  capable  of 
relaying  clear  and  secure  voice  communications  and  secure  data  as  required.  The  payload  is 
capable  of  relaying  voice  communications  between  two  ground  stations  located  50  nautical  miles 
apart  as  required. 

The  RQ-21  A  provides  tactically  useful  target  locations.  While  the  CPD  does  not  specify 
a  threshold  value  for  sensor  point  of  interest  accuracy,  Marine  Corps  guidance  indicates  that 
100-meter  accuracy  is  sufficient  to  support  tactical  operations.  RQ-21  A  provides  a  90  percent 
circular  error  probable  target  location  error  of  43.8  meters.  The  RQ-21  A  target  locations  provide 
Category  IV  accuracy  as  defined  in  Joint  Publication  3-09.3,  Close  Air  Support  (31  to  90 
meters).  Such  accuracy  is  sufficient  to  support  targeting  in  a  conventional  linear  battlefield, 
where  less  target  location  accuracy  is  required  for  successful  target  engagement.  This  accuracy 
does  not  support  targeting  in  a  dense  urban  environment  that  requires  more  accurate  target 
locations. 

Air  vehicle  endurance  meets  the  10-hour  key  performance  parameter.  Nine  flights  during 
IOT&E  exceeded  10  hours  (COMOPTEVFOR  did  not  schedule  every  flight  for  10-plus  hours). 
The  longest  IOT&E  flight  lasted  12.3  hours.  The  longest  flight  during  the  Early  Operational 
Capability  deployment  lasted  12.6  hours,  with  48  of  121  flights  lasting  longer  than  10  hours. 

The  modular  air  vehicle  design  allows  for  payload  changes  well  within  the  60-minute 
Modular/Interchangeable  Payload  key  performance  parameter  threshold.  The  air  vehicle  carries 
payloads  in  one  of  two  modules:  the  nose  module  or  the  Center  of  Gravity  (CG)  Bay.  The 
electro-optical/infrared  sensor  mounts  to  the  nose  module.  The  CG  Bay  carries  the 
communications  relay  payload  and  the  automatic  identification  system.  Build-up  of  an  entire  air 
vehicle  takes  only  20  minutes. 
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The  recessed,  nose-mounted  electro-optical/infrared  payload  requires  circular  orbits  over 
top  of  the  target  to  maintain  continuous  coverage  and  positive  target  identification.  The  use  of 
offset  orbits  results  in  the  fuselage  blocking  the  payload  field  of  view  for  significant  periods  of 
time.  The  actual  duration  of  the  payload  obstruction  ranged  from  5  to  30  seconds,  depending 
upon  air  vehicle  altitude,  bank  angle,  and  distance  from  the  target.  During  IOT&E,  ground  units 
directed  RQ-21 A  operators  to  use  offset  orbits  for  28  of  32  missions.  These  offset  orbits  resulted 
in  auto-track  break  locks  and  loss  of  positive  identification  of  high-value  targets.  There  are  orbit 
shapes  that  would  allow  RQ-21  A  operators  to  maintain  continuous  coverage  of  a  target,  but  the 
current  RQ-21  A  operating  system  limits  operators  to  circular  orbits. 

The  medium- wave  infrared  sensor  does  not  support  search  operations  in  a  maritime 
environment.  Looking  at  a  large,  relatively  uniform  temperature  body  such  as  the  ocean  hinders 
the  sensor’s  auto-focus  capability,  resulting  in  image  blurring  and  haloing. 

The  RQ-21  A  program  office  addressed  the  cybersecurity  vulnerabilities  of  the  RQ-21  A 
electronics  architecture  identified  during  the  first  cybersecurity  adversarial  assessment  at 
Twentynine  Palms  before  the  second  adversarial  assessment  on  the  USS  Anchorage.  During  the 
second  assessment  on-board  the  USS  Anchorage ,  new  vulnerabilities  were  discovered.  The 
classified  appendix  to  this  report  contains  findings  related  to  cybersecurity  testing  and  the 
accompanying  recommendations. 

Operational  Suitability 

The  RQ-21  A  is  not  operationally  suitable.  The  RQ-21  A  demonstrated  an  MFHBAsys,  of 
15.2  hours  versus  the  50-hour  requirement  (80  percent  confidence  interval:  10.6-22.5).  Because 
of  air  vehicle  reliability,  overall  system  availability  did  not  meet  the  80  percent  key  performance 
parameter  threshold  (demonstrated  value  =  66.9  percent).  The  average  service  life  of  the 
propulsion  modules  was  48.9  hours,  which  does  not  meet  the  manufacturer’s  stated  100-hour 
capability. 

Given  that  the  CPD  does  not  provide  a  rationale  for  the  50-hour  MFHBAsys  threshold 
requirement  discussed  above  (which  includes  all  aborts  -  Pre-flight  and  In-flight  aborts), 

DOT&E  chose  to  assess  reliability  against  the  probability  of  completing  a  12-hour  continuous 
coverage  mission  without  experiencing  an  in-flight  aborting  failure  that  causes  the  air  vehicle  to 
return  to  base  prematurely.  Once  launched,  the  RQ-21  A  demonstrated  a  mean  time  between 
in-flight  aborting  failures  resulting  in  a  premature  return  to  base  of  46.7  hours.  This  translates  to 
a  75  percent  probability  of  completing  a  12-hour  on-station  period  without  an  in-flight  aborting 
failure  (80  percent  confidence  interval:  59-87). 

Although  there  are  no  Navy  reliability  requirements  for  failures  rates,  evaluation  of  the 
failures  provides  insight  into  the  future  maintenance  and  logistics  burden  to  the  Navy  for  the 
system.  RQ-21  A  system  demonstrated  a  Mean  Flight  Hours  between  Failure  for  the  system 
(MFHBFsystem)  of  4.6  hours.  The  Mean  Time  between  Failures  for  the  surface  components 
(MTBFsurface  Components),  which  includes  the  ground  control  station,  launcher,  and  recovery  system, 
is  31.3  hours  in  the  land  configuration  and  14.6  hours  in  the  sea  configuration.  The  air  vehicle 
demonstrated  MFHBF  is  5.9  hours. 


v 


The  propulsion  module  unit  has  a  manufacturer-stated  service  life  of  100  operating  hours. 
Upon  reaching  100  hours,  maintainers  replace  the  propulsion  module  unit  and  return  the 
high-time  unit  to  the  manufacturer  for  refurbishment.  Over  the  course  of  995  flight  hours,  the 
Early  Operational  Capability  detachment  replaced  26  propulsion  modules  for  a  demonstrated 
average  life  span  of  48.9  hours  (median  42.2  hours). 

The  RQ-21A  demonstrated  an  availability  rate  of  66.9  percent  (148.5  down  hours  during 
449  test  hours)  during  operations  at  Twentynine  Palms.  The  demonstrated  availability  does  not 
meet  the  80  percent  Operational  Availability  key  performance  parameter.  Air  vehicle  reliability 
caused  148  of  the  148.5  total  system  down  hours.  During  those  148  hours,  the  system  did  not 
have  the  minimum  of  two  air  vehicles  in  a  mission-capable  status.  The  STUAS  Recovery 
System  accounted  for  the  remaining  0.5  hours  of  system  down  time.  The  IOT&E  test  unit 
required  an  influx  of  spare  parts  on  February  6,  2014,  to  complete  that  phase  of  testing.  Without 
the  influx  of  spare  parts,  overall  system  availability  would  have  been  less  than  the  demonstrated 
66.9  percent. 

The  RQ-21A  Naval  Air  Training  and  Operations  Standardization  manual  are  missing 
important  information  regarding  mission  computer  logic.  This  lack  of  information  is  especially 
critical  during  emergencies  where  operators  are  unaware  of  which  conditions  enable/disable 
various  aspects  of  air  vehicle  functionality.  This  lack  of  system  operations  information 
contributed  to  the  loss  of  Air  Vehicle  1010  during  the  first  IOT&E  flight. 

To  support  shipboard  operations,  the  Navy  permanently  installed  some  RQ-21 A  ground 
components  (antennae  interface  modules,  data  link  antennae)  on  selected  ships.  The  ships’ 
personnel,  and  not  the  RQ-21  A  detachment,  own  and  maintain  these  components.  During 
IOT&E,  none  of  the  ships’  personnel  received  training  on  maintaining  these  installed 
components.  The  ship  did  not  receive  spare  parts,  maintenance  manuals,  or  wiring  diagrams 
with  which  to  facilitate  repairs. 

Field  service  representatives  are  required  to  conduct  maintenance  in  accordance  with  the 
Naval  Aviation  Maintenance  Program  (Commander  Naval  Air  Forces  Instruction  4790.2A). 
Instead  of  using  the  established  Naval  Aviation  Logistics  Command  Management  Information 
System  to  track  maintenance  and  flight  information,  the  field  service  representatives  use  a 
program  called  Sapphire.  At  the  completion  of  testing,  operational  testers  conducted  a 
comparison  of  paper  maintenance  action  forms  completed  by  Marines  to  the  Sapphire  database. 
Of  the  136  unscheduled  maintenance  actions  completed  over  the  course  of  IOT&E,  Sapphire 
accurately  archived  15.  Because  of  these  inaccuracies,  the  Marine  Corps  will  not  be  able  to 
determine  the  true  maintenance  burden  associated  with  RQ-21  A  operations. 

Extended  logistics  delay  times  and  production  quality  control  issues  contributed  to  the 
system’s  poor  reliability  and  availability.  In  six  instances,  air  vehicles  spent  time  in  a  non¬ 
mission  capable  status  while  awaiting  spare  parts.  Incorrectly  assembled/configured  components 
increase  maintenance  time  to  repair  or  replace  components,  resulting  in  reduced  mission 
availability. 
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RQ-21A  support  equipment  does  not  facilitate  expeditionary  operations.  As  fielded,  each 
RQ-21A  system  includes  two  tents  used  as  shelter  for  the  ground  control  stations.  Three  module 
containers  for  each  air  vehicle  are  supposed  to  provide  weather  protection  between  flights.  The 
system  does  not  provide  for  inclement  weather  protection  for  support  equipment  and  spare  parts. 
During  eleven  days  of  operations  at  Camp  Lejeune,  maintainers  discovered  six  instances  where 
storage  containers  failed: 

•  On  two  occasions,  foam  cushion  inserts  separated  from  the  wing  container.  This 
allowed  two  wings  to  come  in  contact  and  break  winglet  bolts. 

•  The  design  of  the  fuselage  container  allows  stacking  two  high.  On  one  occasion, 
stacking  containers  caused  a  6-inch  hole  in  the  bottom  container. 

•  After  overnight  rain,  maintainers  discovered  approximately  16  ounces  of  water  on  the 
bottom  of  two  wing  containers  and  one  fuselage  container. 

All  RQ-21A  associated  vehicles  and  trailers  received  certification  for  transport  by 
CH-53E  helicopters,  as  required  by  the  air  transportability  key  performance  parameter.  All 
trailers  meet  the  weight  and  structural  limits  to  allow  for  towing  behind  High  Mobility 
Multipurpose  Wheeled  Vehicles  (HMMWVs),  as  required  by  the  ground  transportability  key 
performance  parameter. 

The  Marine  Corps  intends  that  the  planned  ground  transport  configuration  (four 
HMMWVs,  two  cargo  trailers,  STUAS  Recovery  System  trailer,  and  launcher  trailer)  would 
provide  a  fully  functional  RQ-21A  detachment  to  support  expeditionary  operations.  As  currently 
configured,  the  expeditionary  configuration  relies  upon  significant  help  from  supported  units  to 
conduct  flight  operations.  The  expeditionary  detachment  does  not  include  a  Marine 
communications  specialist  or  dedicated  generators  to  power  the  ground  control  stations. 

Recommendations 

The  Navy  and  Marine  Corps  should  consider  the  following  recommendations  in  order  to 
improve  operational  effectiveness  and  operational  suitability  before  entering  full-rate  production 
and  should  verify  the  corrections  to  deficiencies  during  follow-on  test  and  evaluation.  The  main 
body  of  this  report  contains  additional  recommendations  not  presented  below. 

Operational  Effectiveness 

•  Improve  sensor  resolution  to  increase  probability  of  correct  classification  for  1  -meter 
sized  targets. 

•  Increase  the  number  of  programmed  air  vehicle  loiter  patterns  to  increase  tactical 
flexibility  and  reduce  fuselage  obstruction  of  the  payload. 

•  Fully  integrate  the  communications  relay  payload  into  the  air  vehicle  architecture  to 
allow  for  in-flight  frequency  changes  and  altering  of  other  radio  settings  to  increase 
tactical  utility. 

•  Implement  the  cybersecurity  recommendations  in  the  classified  annex  to  improve 
system  security. 
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Operational  Suitability 


Expand  the  systems  description  and  flight  characteristics  section  of  the  RQ-21A 
Naval  Air  Training  and  Standardization  Procedures  manual  to  allow  operators  to 
safely  react  to  system  emergencies. 

Increase  propulsion  module  service  life  and  reliability  to  reduce  maintainer  workload, 
the  number  of  spares  required,  and  operating  costs. 

Increase  production  quality  control  and  implement  thorough  acceptance  procedures 
for  delivered  systems  and  spare  parts  to  reduce  the  number  of  faulty  items  received  by 
Fleet  operators. 

Fully  train  and  provide  ship  personnel  with  technical  manuals,  wiring  diagrams,  and 
spare  parts  related  to  RQ-21A  shipboard  components  to  increase  RQ-2A1  full 
mission  capability. 

Require  field  service  representatives  to  utilize  the  Naval  Aviation  Logistics 
Command/Management  Information  System  to  track  system  maintenance  to  provide 
the  Marine  Corps  with  better  maintenance  burden  data. 

Increase  expenditures  on  spare  parts  to  reduce  administrative  logistics  delay  times. 
Increase  the  number  of  spare  parts  in  the  pack-up  kit  to  increase  system  availability. 


J.  Michael  Gilmore 
Director 
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Section  One 

System  Overview 


This  document  reports  on  the  evaluation  of  test  adequacy,  operational  effectiveness,  and 
operational  suitability  of  the  RQ-21A  Small  Tactical  Unmanned  Aircraft  System  (STUAS) 
during  Initial  Operational  Test  and  Evaluation  (IOT&E).  The  Navy’s  Commander,  Operational 
Test  and  Evaluation  Force  (COMOPTEVFOR),  with  assistance  from  the  Marine  Corps 
Operational  Test  and  Evaluation  Activity  (MCOTEA),  conducted  the  RQ-21A  IOT&E  between 
January  and  December  2014  in  three  phases  from  three  different  locations.  The  test  locations  in 
sequence  were  Marine  Corps  Air-Ground  Combat  Center  Twentynine  Palms,  California;  Marine 
Corps  Base  Camp  Lejeune,  North  Carolina;  and  aboard  USS  Anchorage.  The  Navy  and  Marine 
Corps  plan  to  use  the  results  of  IOT&E  to  inform  the  Services’  Full-Rate  Production  decision, 
originally  planned  for  October  2015,  now  scheduled  to  occur  in  September  2016.  The  RQ-21A 
is  an  Acquisition  Category  II  program. 

Mission  Description  and  Concept  of  Employment 

The  Services  intend  the  RQ-21A-equipped  detachment  to  provide  an  expeditionary, 
organic  Intelligence,  Surveillance,  and  Reconnaissance  (ISR)  asset  at  the  tactical  level  to  support 
both  maritime  and  shore-based  operations  within  the  respective  tactical  commander’s  area  of 
responsibility.  Mission  support  areas  include: 

•  target  development  (to  include  locating  and  tracking), 

•  maritime  and  shore  target  identification, 

•  near-shore  and  beach  landing  site  and  helicopter  landing  zone 
reconnaissance/surveillance, 

•  area/point/route  reconnaissance, 

•  vehicle  convoy  over-watch, 

•  riverine  and  small  craft  transit  over-watch, 

•  maritime/land  direct  action  over- watch. 

The  Marines  intend  task-organized  detachments  from  Marine  Unmanned  Aerial  Vehicle 
Squadrons  (VMU)  to  operate  and  maintain  RQ-21A  in  support  of  any  sized  Marine  Air-Ground 
Task  Force.  Each  VMU  will  possess  equipment  and  manpower  sufficient  to  support  nine 
RQ-21A  detachments.  Detachments  deploy  with  Marine  Expeditionary  Units  (MEUs)  and 
regimental  combat  teams.  To  support  an  MEU,  the  VMU  detachment  will  integrate  with  the 
MEU’s  aviation  combat  element  during  the  pre-deployment  training  cycle,  embark  the  requisite 
personnel  and  equipment  aboard  an  L-class  ship,  and  conduct  flight  operations  in  the  maritime 
domain. 

If  required  to  go  from  ship  to  shore,  the  VMU  detachment  will  disembark  the  requisite 
personnel  and  equipment  on  CH-53E  helicopters  and  surface  landing  craft,  transition  ashore, 
deploy  to  and  establish  a  suitable  expeditionary  land-based  site  (usually  co-located  with  the 
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supported  unit  headquarters),  to  commence  flight  operations  in  support  of  the  ground 
commander’s  objectives.  The  VMU  detachment  may  conduct  “hub-and-spoke”  operations 
during  which  a  launch,  recovery,  and  maintenance  “hub”  capability  remains  afloat  while  a 
mission  control  “spoke”  capability  goes  ashore  in  order  to  extend  the  range  at  which  the  RQ-21A 
can  support  MEU  objectives  inland  with  a  minimal  footprint  ashore. 

During  land-based  operations,  the  VMU  detachment  must  be  able  to  transport  requisite 
personnel  and  equipment  via  High  Mobility  Multipurpose  Wheeled  Vehicle  (HMMWV)  to 
maneuver  as  required  in  support  of  ground  units. 


System  Description 

As  depicted  in  Figure  1-1,  the  RQ-21 A  system  includes  five  air  vehicles;  each  air  vehicle 
is  equipped  with  an  electro-optical/infrared  payload  and  a  communications  relay  payload,  two 
ground  control  stations  (each  ground  control  station  has  two  operator  workstations),  four  ground 
data  terminals,  a  launcher,  a  recovery  system,  and  government- furnished  vehicles  and  support 
systems.  The  Pack-Up  Kit  provides  spare  components  and  a  tool  kit. 


Pack-Up  Kit 
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Government  Furnished  Equipment 


Figure  1-1.  RQ-21A  Small  Tactical  Unmanned  Aircraft  System 


The  air  vehicle  is  a  modular  system,  composed  of  small  easily  assembled  line  replaceable 
units.  It  has  a  multi-function  ball  turret  that  houses  mission  sensors,  and  various  connection 
points  and  bays  for  custom  payloads  (see  Figure  1-2).  The  air  vehicle  is  capable  of  taking  off 
and  landing  without  a  runway  via  the  launch  and  recovery  systems.  The  air  vehicle  is  powered 
by  a  two-cylinder,  two-stroke,  100  cc  electronically  fuel-injected  engine  that  operates  on  JP-5 
(military  aviation  jet  fuel)  and  equivalent  fuel  types  (with  the  addition  of  a  special  synthetic 
lubricating  oil  additive). 
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Air  Vehicle 

Weight 

Gross  Takeoff  Weight 

135  pounds 

Dry  weight 

110  pounds 

Fuel 

25  pounds 

WingSpan 

16.25  feet 

Length 

7.2  feet 

Figure  1-2.  RQ-21A  Air  Vehicle 


The  ground  control  station  (GCS)  is  used  to  plan  missions,  control  and  monitor  the  air 
vehicle,  and  manage  data  received  from  the  air  vehicle.  Figure  1-3  shows  the  tabletop  ground 
control  station  configuration  used  during  operations  at  Marine  Corps  Air-Ground  Combat  Center 
Twentynine  Palms.  The  GCS  includes  two  operator  workstations,  two  ground  data  terminals  for 
air  vehicle  command  and  control  and  video  downlink,  and  a  Global  Positioning  System  (GPS) 
Electronics  Module.  Air  vehicle  operators  control  the  air  vehicle  and  multi-function  ball  turret 
through  the  GCS. 


Figure  1-3.  Tabletop  Ground  Control  Station  Configuration 
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Each  operator  workstation  consists  of  four  modules:  the  electronics  power  module, 
network  computer  module,  user  interface  module,  and  the  data  storage  module  (see  Figure  1-4). 


Data  Storage  Module 


User  Interface  Module 


Network  Computer  Module 
Electronics  Power  Module 


Figure  1-4.  Operator  Workstation 


The  RQ-21A  launcher  is  a  pneumatically-controlled  device  that  accelerates  the  air 
vehicle  to  flight  speed  at  takeoff  and  enables  expeditionary  employment  of  the  RQ-21 A  in 
austere  conditions  without  runways  (see  Figure  1-5).  The  launcher  system  is  self-powered  and 
designed  for  transport  by  air  (via  CH-53  helicopter,  C-130  aircraft,  and  C-17  aircraft)  or  by  land 
(towed  by  HMMWV). 


Figure  1-5.  The  RQ-21A  Launcher 
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The  RQ-21 A  recovery  system  employs  a  telescoping  hydraulic  mast  and  boom  system 
that  suspends  a  rope  to  capture  the  air  vehicle  in  mid-flight  (see  Figure  1-6).  After  the  rope 
captures  the  air  vehicle,  the  air  vehicle  spins  about  the  rope  down  to  the  ground.  The  recovery 
system  mounts  on  a  modified  tactical  Military  Standard  (MIL-STD)  trailer  chassis  with  an 
engine-driven  hydraulic  power  unit.  External  electrical  power  and  fiber  optic  signal  cables  are 
required  to  support  the  GPS  electronics  module  onboard  the  RQ-21  A  recovery  system. 


Figure  1-6.  RQ-21A  Recovery  System 


Each  RQ-21  A  system  has  two  Tactical  ISR  Processing,  Exploitation,  and  Dissemination 
Systems  (TIPS)  that  serve  as  the  cross-domain  solution  from  the  unclassified  GCS  to  any 
classified  network.  The  TIPS  allows  intelligence  analysts  to  monitor  near  real-time  RQ-21  A 
imagery  and  create  intelligence  products  for  use  by  supported  units.  The  TIPS  provides  the 
conduit  for  streaming  live  RQ-21  A  imagery  across  classified  networks  for  viewing  by  any  user 
with  access  to  that  particular  network. 

The  Marine  Corps  expects  each  detachment  to  operate  and  maintain  the  RQ-21  A  for  up 
to  12  hours  a  day  with  a  surge  rate  of  24  hours  a  day  for  a  10-day  period.  Additional  Marines 
may  be  required  to  sustain  surge  operations  and  normal  operations  for  extended  periods  of  time. 
The  VMU  commander  will  tailor  actual  detachment  sizes  based  on  the  expected  mission  and 
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duration  of  operations.  This  arrangement  is  typical  for  all  Marine  Air-Ground  Task  Force 
operations. 

In  accordance  with  Marine  Corps  planning,  the  Navy’s  program  manager  for  Navy  and 
Marine  Corps  Small  Tactical  Unmanned  Air  Systems  (PMA-263)  has  funded  for  contractor 
maintenance  support  to  the  VMUs  and  all  detachments  until  two  years  after  the  Marine  Corps 
declares  RQ-21A  Initial  Operating  Capability,  which  is  currently  expected  in  mid  FY16.  Four 
field  service  representatives  will  deploy  with  each  RQ-21A-equipped  Marine  Expeditionary 
Unit. 

Organizational  level  maintenance  is  limited  to  the  removal  and  replacement  of  individual 
modules  for  the  air  vehicles  and  operator  workstations.  Units  return  removed  modules  to  the 
manufacturer  for  repair. 
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Section  Two 

Test  Adequacy 


Operational  testing  of  the  RQ-21A  was  adequate  to  support  an  evaluation  of  operational 
effectiveness  and  operational  suitability.  The  Navy’s  Commander,  Operational  Test  and 
Evaluation  Force  (COMOPTEVFOR)  conducted  the  testing  in  accordance  with  a  DOT&E- 
approved  test  plan.  The  Navy  and  Marine  Corps  will  use  the  results  of  Initial  Operational  Test 
and  Evaluation  (IOT&E)  to  support  a  Full-Rate  Production  decision,  originally  planned  for 
October  2015,  now  scheduled  to  occur  in  September  2016. 

Operational  Testing 

The  operational  test  unit  consisted  of  Sailors  and  Marines  from  three  units.  Marine 
Unmanned  Aerial  Vehicle  Squadron  2  (VMU-2)  provided  air  vehicle  operators  and  maintainers, 
Marine  Operational  Test  and  Evaluation  Squadron  22  (VMX-22)  provided  air  vehicle  operators, 
and  Air  Test  and  Evaluation  Squadron  ONE  (VX-1)  provided  a  maintainer  and  an  air  vehicle 
operator.  Four  Insitu  (the  RQ-21A  prime  contractor)  field  service  representatives  provided 
technical  and  maintenance  support  in  a  manner  consistent  with  the  Navy  fielding  plan.  VX-1 
Operational  Test  Directors  provided  day-to-day  control  of  test  events.  The  Marine  Corps 
Operational  Test  and  Evaluation  Activity  assisted  with  IOT&E  planning,  execution,  and 
reporting. 

COMOPTEVFOR  conducted  testing  in  three  phases  between  January  and  December 
2014,  as  shown  in  Table  2-1. 


Table  2-1.  Phases  of  Initial  Operational  Test  and  Evaluation 


Test 

Phase 

Dates 

Location 

Flights 

Flight  Hours 
(Flown/Planned) 

Phase  1 

January  15  - 
February  1 3 

Marine  Corps  Air-Ground 
Combat  Center 

26 

188.3/324 

Phase  la 

June  4-14 

Marine  Corps  Base  Camp 
Lejeune 

8 

20.9/NA 

Phase  2 

December  1-10 

USS  Anchorage 
(LPD  23) 

4 

24.4/24 

Total 

38 

233.6/348 

During  Phase  1,  the  RQ-21A-equipped  unit  supported  a  Marine  Corps  Integrated 
Training  Exercise.  Missions  flown  during  Phase  1  included  communications  relay,  convoy 
over-watch,  tactical  recovery  of  aircraft  and  personnel,  and  various  reconnaissance  and 
surveillance  missions.  An  air  vehicle  mission  computer  failure  during  the  first  flight  of  Phase  1 
resulted  in  the  destruction  of  an  air  vehicle.  Post-mishap  investigations  suspended  operational 
test  flights  for  10  days.  The  first-flight  mishap  and  reliability  issues  throughout  Phase  1  limited 
the  RQ-21A  detachment  to  flying  188.3  of  the  324  planned  flight  hours. 
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Because  of  the  reduced  number  of  flight  hours,  overall  poor  system  performance  during 
Phase  1,  and  a  change  to  air  vehicle  software,  COMOPTEVFOR  added  Phase  la,  a  second 
land-based  phase  of  test  not  contained  in  the  DOT&E-approved  test  plan.  RQ-21A  operations 
during  this  period  supported  a  forward  air  controller  training  exercise  and  repeated  collection  of 
target  location  data  points  because  of  the  software  changes  made  to  the  system  post-Phase  1 
testing. 

Operational  testing  concluded  with  the  ship-based  Phase  2.  During  this  phase  of  test,  the 
RQ-21A-equipped  unit  assessed  shipboard  compatibility  for  maintainability  and  flight 
operations.  Missions  included  support  for  visit,  board,  search,  and  seizure  operations,  and 
provided  full-motion  video  of  surface  contacts  to  the  ship’s  Combat  Information  Center. 

Cybersecurity 

The  two  RQ-21A  cybersecurity  test  events  executed  during  IOT&E  were  adequate  to 
assess  cybersecurity  vulnerabilities.  The  Marine  Corps  Information  Assurance  Red  Team 
conducted  testing  on  the  land-based  system  during  operations  at  Twentynine  Palms.  This  period 
assessed  operator  workstations  and  the  Tactical  Intelligence,  Surveillance,  and  Reconnaissance 
(ISR)  Processing,  Exploitation,  and  Dissemination  System  (TIPS)  against  the  insider  threat. 

COMOPTEVFOR,  working  with  the  Navy  Information  Operational  Command-Norfolk, 
conducted  cybersecurity  testing  of  the  sea-based  system  aboard  USS  Anchorage  (LPD  23).  This 
test  period  assessed  the  operator  workstations  and  TIPS,  in  addition  to  the  antenna  interface 
module,  the  maintenance  laptop,  and  the  air  vehicle.  Testers  attempted  to  penetrate  the  RQ-21A 
system  externally  through  the  Fleet  Network  Operations  Center,  Automated  Digital  Network 
System,  and  Shipboard  Wide  Area  Network.  The  air  vehicle  remained  grounded  during 
cybersecurity  testing  for  flight  safety  reasons.  Figure  2-1  shows  the  RQ-21A  cybersecurity 
architecture. 


AIM  -  Antenna  Interface  Module 

DSM  -  Data  Storage  Module 

GEM  -  GPS  Electronic  Module 

OWS  -  Operator  Workstation 

SIPRnet  -  Secret  Internet  Protocol  Router  Network 

TIPS  -  Tactical  Intelligence,  Surveillance  and 

Reconnaissance  Processing,  Exploitation  and  Dissemination 
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Figure  2-1.  RQ-21A  Architecture 
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Additional  Data 

In  May  2014,  the  Marine  Corps  deployed  an  Early  Operational  Capability  RQ-21A 
system  to  support  overseas  combat  operations.  This  deployed  RQ-21A  detachment  consisted  of 
up  to  eight  air  vehicles,  two  launchers,  and  two  recovery  systems.  Between  May  and  September, 
the  system  flew  approximately  121  flights  for  995  flight  hours.  Each  day  the  detachment  sent  a 
daily  summary  containing  the  status  of  each  major  piece  of  equipment  and  a  narrative  summary 
of  the  day’s  flight  activities.  Data  collected  by  the  Early  Operational  Capability  detachment 
while  deployed  overseas  provided  additional  insights  into  system  reliability  and  availability. 

After  experiencing  communications  relay  payload  difficulties  during  the  Camp  Lejeune 
phase  of  IOT&E,  COMPTEVFOR  and  the  RQ-21A  program  office  sent  personnel  to  observe 
two  periods  of  VMU-2  flight  operations  in  the  vicinity  of  Marine  Corps  Air  Station  Cherry  Point, 
North  Carolina.  Over  two  separate  weeks,  observers  identified  the  cause  of  the  communications 
relay  problems  and  updated  technical  publications  to  ensure  future  repeatability  in  the  payload 
programming  and  employment  process. 

Test  Limitations 

There  were  no  test  limitations  affecting  the  scope  of  DOT&E’s  assessment. 
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Section  Three 

Operational  Effectiveness 

The  RQ-21A  is  not  operationally  effective.  The  detachment  equipped  with  RQ-21A  is 
not  effective  in  supporting  the  ground  commander’s  mission  because  of  an  inability  to  have  an 
unmanned  aircraft  arrive  on  station  at  the  designated  time  and  remain  on  station  for  the  duration 
of  the  tasked  period.  The  electro-optical/infrared  sensor  provides  accurate  target  locations. 
Operators  using  the  sensors  were  not  able  to  correctly  classify  1 -meter  targets,  which  is 
necessary  for  consistent  classification  of  hostile/non-hostile  intent  on  the  part  of  individuals  by 
classifying  such  items  as  rifles,  rocket-propelled  grenade  launchers,  and  shovels. 

Mission  Accomplishment 

Mission  accomplishment  begins  with  the  unmanned  air  vehicle  arriving  at  the  desired 
on-station  location,  at  the  prescribed  time,  and  establishing  contact  with  the  supported  unit.  The 
unmanned  aircraft  should  then  be  capable  of  performing  assigned  tasks  for  the  duration  of  the 
required  on-station  time.  Air  vehicles  that  arrive  late  or  depart  early  may  imperil  ground  forces 
and  adversely  affect  mission  accomplishment. 

Once  on  station,  the  air  vehicle’s  sensors  should  support  the  commander’s  intelligence 
requirements  by  providing  accurate  locations  and  reports  of  enemy  actions.  The  unmanned 
aircraft  should  support  additional  missions  for  which  it  is  equipped,  such  as  communications 
relay,  and  the  unmanned  aircraft  should  have  the  capability  of  accepting  dynamic  retasking  to 
support  the  commander’s  emergent  needs.  The  mission  concludes  with  the  safe  recovery  of  the 
air  vehicle  without  encountering  a  significant  amount  of  damage. 

Mission  Coverage 

The  RQ-21A  does  not  provide  commanders  with  a  timely  and  dependable  intelligence 
asset.  This  metric  provides  an  indication  of  the  level  of  support  a  ground  commander  can  expect 
from  the  RQ-21A  system.  The  Mission  Coverage  metric  measures  the  percentage  of  time  an  air 
vehicle  is  on  station  when  tasked  to  do  so.  During  the  Initial  Operational  Test  and  Evaluation 
(IOT&E),  the  RQ-21A  equipped  unit  provided  coverage  during  68  percent  of  the  tasked  on- 
station  hours  (83.8  of  122.7  hours,  see  left  side  of  Figure  3-1).  This  means  RQ-21A  was  absent 
or  not  capable  of  providing  requisite  support  one-third  of  the  time  the  commander  expected  to 
have  support.  Losses  in  on-station  time  resulted  from  the  following: 

•  RQ-21 A  failed  to  support  20. 1  hours  (16.4  percent)  of  the  on-station  hours  because  of 
in-flight  aborting  failures  that  caused  an  early  return  to  base. 

•  Pre-flight  aborting  failures  that  delayed  or  cancelled  launches  resulted  in  the  loss  of 
10.7  hours  (8.7  percent)  of  the  on-station  flight  hours. 

•  In-flight  troubleshooting  of  sensor  anomalies  caused  the  loss  of  the  final  8.1  hours 
(6.6  percent)  of  on-station  hours. 

RQ-21  A  flew  77  percent  of  scheduled  flights  hours  (233.6  of  301.7,  see  right  side  of 
Figure  3-1).  Unlike  the  Mission  Coverage  metric,  this  number  considers  all  flight  hours 
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scheduled  across  the  entirety  of  the  IOT&E.  This  includes  the  tasked  on-station  times,  transit 
times  to  and  from  the  on-station  location,  training  flights,  and  flights  to  collect  specific 
operational  test  data  points.  In  this  case,  pre-flight  aborts  caused  the  greatest  loss  of  33.9  flight 
hours  (1 1.2  percent),  followed  by  in-flight  aborting  failures,  which  caused  a  loss  of  23.4  hours 
(7.7  percent),  and  in-flight  troubleshooting,  which  caused  a  loss  10.7  hours  (3.6  percent). 


Reason  for  Lost  Hours 

Airborne  (No  Lost  Hours) 

|  In-Flight  Abort 
|  Pre-Flight  Abort 
Troubleshooting 


Figure  3-1.  Mission  Coverage 


Mission  Completion  Rate 

The  RQ-21A  demonstrated  a  low  probability  of  successfully  completing  realistic 
missions  during  IOT&E.  As  shown  in  Figure  3-2,  19  of  45  attempted  flights  (42  percent) 
launched  on  time,  supported  assigned  tasking,  and  remained  on  station  for  the  duration  of  the 
assigned  period.  Of  the  45  attempted  launches,  35  (78  percent)  provided  some  level  of  utility  to 
ground  commanders  (launched  at  some  point  and  provided  some  on-station  time). 
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Figure  3-2.  Mission  Completion  Rate 


Target  Classification 

The  RQ-21A  sensor  does  not  meet  one  of  the  two  classification  key  performance 
parameters  (KPPs).  The  Capability  Production  Document  (CPD)  establishes  a  classification 
KPP  for  the  electro-optical  sensor  and  one  for  the  infrared  sensor.  Each  classification  KPP 
specifies  a  50  percent  probability  of  classifying  an  object  at  an  air  vehicle-to-target  slant  range  of 
4,242  feet  (3,000  feet  above  ground  level  with  a  sensor  depression  angle  of  45  degrees).  The 
electro-optical  sensor  is  required  to  support  classification  of  1 -meter  sized  linear  objects 
(weapons  or  tools)  while  the  infrared  sensor  is  required  to  support  classification  of  3 -meter  sized 
linear  objects  (vehicle  chassis  type). 

Using  design  of  experiments,  the  Navy’s  Commander,  Operational  Test  and  Evaluation 
Force  (COMOPTEVFOR)  developed  a  data  collection  plan  to  characterize  target  classification 
capabilities  for  each  sensor  across  a  range  of  air  vehicle-to-target  distances  and  sensor  depression 
angles.  As  shown  in  Table  3-1,  test  results  indicate  that  the  electro-optical  sensor  does  not  meet 
the  50  percent  probability  threshold  for  correct  classification  of  a  1 -meter  object  in  desert  (41 
percent  [80  percent  confidence  interval:  24-60])  or  maritime  (43  percent  [80  percent  confidence 
interval:  24-64])  environments  at  the  specified  slant  range. 

The  infrared  sensor  does  meet  the  50  percent  threshold  probability  for  correctly 
classifying  3 -meter  objects  by  demonstrating  100  percent  correct  classification. 
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Table  3-1.  Percent  Correct  Classification  at  Slant  Range  =  4,242  feet 
(With  80  percent  Confidence  Intervals) 


Sensor 

Environment 

1 -meter  Object 

3-meter  Object 

Electro-optical 

Desert 

41% 

(24,  60) 

Maritime 

43% 

(24,  64) 

3-meter  objects 
had  1 00  percent 
classification. 

Infrared 

Desert 

48% 

(34,  63) 

Maritime 

11% 

(4,  25) 

Infrared  sensor  performance  in  a  maritime  environment  discussed  in  the  Sensor 
Performance  section  below  could  explain  the  poor  probability  of  correct  classification  results  in 
Table  3-1. 

Communications  Relay 

The  communications  relay  payload  limits  the  commander’s  tactical  flexibility  and 
mission  accomplishment.  The  payload  is  capable  of  relaying  clear  and  secure  voice 
communications  and  secure  data  as  required.  The  payload  is  capable  of  relaying  voice 
communications  between  two  ground  stations  located  50  nautical  miles  apart,  as  required. 

The  communications  relay  payload  supported  plain  voice  relay  operations  at  Twentynine 
Palms  between  two  aircraft,  and  between  VMU-1  Shadow  operators  and  Joint  Terminal  Attack 
Controllers  conducting  close  air  support  missions.  Operational  testers  demonstrated  secure  voice 
communications  relay  during  the  two  troubleshooting  periods  in  North  Carolina  and 
demonstrated  secure  data  relay  aboard  USS  Anchorage. 

The  communications  relay  payload  is  constrained  to  a  single  frequency  in  each  of  the  two 
radios,  which  limits  the  commanders’  tactical  flexibility.  Before  launch,  maintainers  pre-set  the 
frequency  in  each  radio.  Once  airborne,  the  only  interface  between  the  air  vehicle  and  the 
communications  relay  payload  is  a  power  connector.  Operators  can  power  the  communications 
relay  payload  on  and  off,  but  cannot  change  frequencies.  This  means  that  the  communications 
relay  payload  supports  pre-planned  communications  relay  missions  on  a  single  pre-set  frequency 
pair. 

In  practice,  when  an  air  vehicle  launches  on  a  mission,  it  may  or  may  not  be  tasked  to 
support  a  pre-planned  communications  relay  mission.  Over  the  course  of  a  10-hour  mission,  the 
air  vehicle  may  receive  several  dynamic  retasking  missions,  one  of  which  might  be 
communications  relay.  If  the  units  requesting  communications  relay  support  are  not  on  the 
single  pair  of  pre-programmed  frequencies  on  the  air  vehicle,  the  supported  units  must  change 
their  communications  plan  to  accommodate  the  air  vehicle’s  frequencies.  Due  to  the  time  and 
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coordination  necessary  for  such  an  undertaking,  this  is  not  a  practical  solution  for  the  supported 
ground  unit. 

Four  examples  from  IOT&E  demonstrate  the  mission  failures  inherent  to  the  single  pair 
of  frequencies: 

•  During  a  tactical  recovery  of  aircraft  and  personnel  mission,  the  communications 
relay  payload  was  not  set  to  the  frequency  used  by  the  survivors  on  the  ground.  The 
RQ-21A  operators  were  not  able  to  monitor  the  survivors’  communications  for 
information  on  their  location. 

•  During  one  mission,  RQ-21A  operators  received  a  tasking  to  support  a  ground  unit  in 
the  search  for  missing  personnel.  The  RQ-21A  operators  were  not  able  to  switch  the 
communications  relay  payload  to  the  search  frequency,  so  they  could  not  contribute 
to  the  search. 

•  During  one  pre-planned  communications  relay  mission,  maintainers  loaded  the  wrong 
combination  of  frequencies  into  two  separate  air  vehicles.  One  air  vehicle  was  to 
support  the  pre-planned  communications  relay  mission  while  the  other  was  to  support 
IOT&E  data  collection.  Because  operators  could  not  change  to  the  correct 
frequencies,  both  air  vehicles  were  required  to  support  the  pre-planned 
communications  relay  mission.  Communications  went  from  Ground  Unit  A,  to  Air 
Vehicle  A,  to  Air  Vehicle  B,  to  Ground  Unit  B  and  back  again. 

•  During  a  forward  air  controller  training  exercise,  controllers  planned  for  the 
communications  relay  payload  to  switch  frequencies  to  support  different  teams  of 
forward  controllers  throughout  each  flight.  In  this  case,  the  communications  relay 
payload  was  inoperable.  Had  the  communications  relay  payload  functioned  properly; 
all  teams  would  have  been  required  to  operate  on  a  single  set  of  frequencies  that 
would  have  delayed  the  exercise. 

The  CPD  does  not  call  for  the  ability  to  change  frequencies  while  airborne. 
COMOPTEVFOR  determined  such  a  capability  to  be  a  derived  requirement  and  sought 
concurrence  from  the  Marine  Corps  Capabilities  Development  Directorate.  The  Director  of  the 
Capabilities  Development  Directorate  did  not  concur  with  COMOPTEVFOR’ s  derived 
requirement  because  they  believe  the  capability  to  change  frequencies  in-flight  is  not 
technologically  mature  and  not  a  present  day  operational  requirement.  The  Capabilities 
Development  Directorate  did  not  incorporate  the  ability  to  change  frequencies  during  flight 
capability  requirement  into  the  CPD. 

The  AN/PRC- 152  radios  used  in  the  RQ-21A  communications  relay  payload  support 
frequency  changes.  Currently,  connections  between  the  communications  relay  payload  and  the 
RQ-21A  air  vehicle  consist  of  a  power  connector  and  two  antennae  connections.  In-flight 
frequency  changes  would  require  the  addition  of  a  data  cable  to  the  AN/PRC- 152  radios  and  a 
user  interface  in  the  Operator  Workstation  software.  Such  technology  is  available  and  mature; 
the  RQ-7  Shadow  communications  relay  package,  which  the  Marine  Corps  employs,  has  had  an 
in-flight  frequency  changing  capability  for  several  years. 
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The  communications  relay  payload  does  not  support  operations  between  ground  units 
employing  Network  Identifiers  (Net  IDs)  that  provide  frequency  hopping  to  protect  against 
communications  jamming.  The  communications  relay  payload  cannot  load  the  required 
Transmission  Security  Key  nor  can  it  maintain  the  required  GPS  Time  Synchronization.  As  a 
result,  the  communications  relay  payload  cannot  support  convoy  over-watch  missions  where  the 
supported  ground  unit  employs  the  frequency  hopping  capability  in  their  normal  combat 
operations. 

Programming  communications  relay  payload  radios  is  a  burdensome  process  consisting 
of  72  steps  (36  for  each  radio).  This  process  proved  cumbersome  to  maintainers  and  is 
exacerbated  by  the  lack  of  formal  training  and  poorly  written  manuals.  Twenty- four  of  these 
steps  contain  multiple  options.  Five  of  the  steps  contain  multiple  options  that  must  be  set  to  one 
specific  option.  One  step  has  an  option  that  must  be  the  same  as  the  supported  ground  unit  radios 
using  the  communications  relay  payload  to  communicate,  requiring  additional  coordination  with 
the  supported  unit  well  prior  to  launching  the  air  vehicle.  Choosing  the  wrong  option  during  the 
programming  process  could  degrade  or  prevent  communications  relay  after  the  air  vehicle 
launches. 

Target  Attack 

The  RQ-21A  provides  tactically  useful  target  locations.  While  the  CPD  does  not  specify 
a  threshold  value  for  sensor  point  of  interest  accuracy,  Marine  Corps  guidance  indicates  that 
100-meter  accuracy  is  sufficient  to  support  tactical  operations.  RQ-21A  provides  a  90  percent 
circular  error  probable  target  location  error  of  43.8  meters  at  3,000  feet  above  ground  level  with 
a  45-degree  sensor  depression  angle. 

RQ-21A  target  locations  provide  Category  IV  accuracy  as  defined  in  Joint  Publication 
3-09.3,  Close  Air  Support  (3 1-90  meters).  Such  accuracy  is  sufficient  to  support  targeting  in  a 
conventional,  linear  battlefield,  where  less  target  location  accuracy  is  required  for  successful 
target  engagement.  This  accuracy  does  not  support  targeting  in  a  dense  urban  environment  that 
requires  more  accurate  target  locations. 

Target  locations  reported  by  RQ-21A  operators  showed  a  systematic  error,  consistently 
reporting  target  locations  as  further  east  (on  average,  19  meters)  and  further  north  (on  average, 

17  meters)  than  actual  locations  (see  Figure  3-3). 
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Figure  3-3.  Sensor  Point  of  Accuracy  Data 

The  RQ-21A  does  provide  a  laser  rangefinder  to  measure  the  air  vehicle-to-target 
distance.  Other  unmanned  aircraft  systems  use  this  information  in  their  target  location 
calculations  to  reduce  location  error.  The  RQ-21 A  does  not  use  laser  rangefinder  information  to 
assist  in  targeting. 

The  infrared  marker  provides  utility  when  used  to  identify  specific  targets  for  ground 
observers.  Successful  employment  of  the  laser  marker  required  RQ-21  A  operators  to  start  by 
putting  the  marker  within  100  meters  of  the  ground  observer’s  position.  The  RQ-21  A  operator 
would  then  walk  the  marker  out  to  the  target.  Ground  observers  could  follow  the  marker  out  to 
500  meters,  at  which  point  they  lost  sight  of  the  marker.  Low  infrared  marker  energy  and 
“micro-terrain”  (convex  and  concave  terrain  features)  made  it  difficult  to  pick  out  the  infrared 
marker  spot  from  background  clutter. 

System  Performance 
Air  Vehicle  Endurance 

Air  vehicle  endurance  meets  the  10-hour  KPP.  An  unfueled  air  vehicle  (with  payloads) 
weighs  110  pounds,  with  a  maximum  gross  weight  of  135  pounds.  With  25  pounds  of  fuel  and 
an  average  2-pound  per  hour  bum  rate,  the  calculated  endurance  is  approximately  12  hours. 
Actual  endurance  depends  upon  the  amount  of  climbing  and  descending  and  high-speed  flight 
required  to  support  mission  tasking.  Frequent  climbs  and  higher  airspeeds  increase  the  fuel  bum 
rate,  thus  decreasing  endurance. 

COMOPTEVFOR  did  not  schedule  every  IOT&E  flight  to  last  ten  or  more  hours. 
Mission  tasking,  training  requirements,  and  data  collection  drove  actual  flight  durations.  Of  the 
38  IOT&E  flights,  nine  exceeded  10  hours.  The  longest  IOT&E  flight  lasted  12.3  hours.  The 


17 


longest  Early  Operational  Capability  deployment  flight  lasted  12.6  hours,  with  48  flights  lasting 
longer  than  10  hours. 

Payload  Modularity 

The  modular  air  vehicle  design  allows  for  payload  changes  within  the  60-minute 
Modular/Interchangeable  Payload  KPP  threshold.  Build-up  of  an  entire  air  vehicle  takes  20 
minutes.  As  shown  in  Figure  3-4,  the  air  vehicle  carries  payloads  in  one  of  two  modules:  the 
nose  module  or  the  Center  of  Gravity  (CG)  Bay. 


Nose 

Module 


Figure  3-4.  Air  Vehicle  Payload  Modules 


The  electro-optical/infrared  sensor  mounts  to  the  nose  module.  The  CG  Bay  carries  the 
communications  relay  payload  and  the  automatic  identification  system.  The  CG  Bay  will  carry 
additional  payloads  in  the  future  provided  they  have  the  appropriate  interface  connections. 

Maintainers  are  able  to  remove  and  replace  the  nose  module  or  the  CG  Bay  well  within 
the  required  60-minute  threshold.  The  nose  module  connects  to  the  air  vehicle  fuselage  with 
eight  bolts  and  has  a  pneumatic  tubing  connection  for  the  pitot  systems  and  a  serial  port  type 
connection  to  the  avionics  module.  The  CG  bay  connects  to  the  air  vehicle  fuselage  with  eight 
bolts  and  a  single  power  connection  to  the  avionics  module. 

Sensor  Performance 

The  recessed  nose-mounted  electro-optical/infrared  payload  requires  circular  orbits  over 
top  of  the  target  to  maintain  continuous  coverage  and  positive  target  identification.  The  use  of 
offset  orbits  results  in  the  fuselage  blocking  the  payload  field  of  view  for  significant  periods  of 
time  (see  Figure  3-5).  The  actual  duration  of  the  payload  obstruction  ranged  from  5  to  30 
seconds  depending  upon  air  vehicle  altitude,  bank  angle,  and  distance  from  the  target.  During 
IOT&E,  these  periods  of  payload  obstruction  resulted  in  auto-track  break  locks  and  loss  of 
positive  identification  of  high-value  targets. 

Battalion  Air  Officers  and  Joint  Tactical  Air  Controllers  directed  RQ-21A  operators  to 
place  the  air  vehicle  in  an  offset  loiter  position  in  28  of  32  missions  in  accordance  with  Joint 
Publication  3-52  Joint  Airspace  Control  and  airspace  control  plans  used  in  overseas  operations. 
Offset  orbits  are  typically  preferred  to  orbits  over  top  of  the  target  because  they  make  the  air 
vehicle  less  visually  and  aurally  detectable  to  the  target.  In  cases  where  the  target  may  be  fired 
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upon,  an  offset  orbit  may  be  necessary  to  ensure  that  the  air  vehicle  will  not  obstruct  the 
projectile’s  approach  to  the  target. 


Figure  3-5.  Sensor  Payload  Fuselage  Obstruction 

As  shown  in  Figure  3-6,  there  are  orbit  shapes  that  allow  RQ-21A  operators  to  maintain 
continuous  coverage  of  a  target  from  an  offset  orbit.  The  so-called  “bow-tie”  orbit  solves  the 
fuselage  blocking  the  payload  issue  while  maintaining  continuous  target  coverage.  As  currently 
designed,  the  RQ-21A  operating  system  limits  operators  to  circular  orbits.  An  experienced  air 
vehicle  operator  was  able  to  execute  a  bow-tie  orbit  by  manually  moving  a  circular  orbit  back 
and  forth  as  the  payload  approached  the  fuselage  blocking  point.  This  manual  bow-tie  pattern 
requires  an  increased  workload  that  prevents  the  operator  from  performing  any  other  task  during 
its  conduct. 
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The  medium- wave  infrared  sensor  does  not  support  search  operations  in  a  maritime 
environment.  Looking  at  a  large,  relatively  uniform  temperature  body  such  as  the  ocean  hinders 
the  sensor’s  auto-focus  capability,  resulting  in  blurring  and  haloing  of  the  image,  as  shown  in 
Figure  3-7. 


Ocean  Water  With  No  Visible  Contrast 


Whitecaps  Provide  Some  Contrast 


Figure  3-7.  Infrared  Sensor  in  Maritime  Environment 


The  left  image  in  Figure  3-7  shows  the  ocean  without  contrast.  Without  contrast,  the 
sensor’s  auto-focus  feature  is  not  able  to  provide  a  clear  image.  The  right  image  in  Figure  3-7 
shows  some  contrast  in  the  form  of  whitecaps  that  allows  for  auto-focus  engagement.  Both 
images  show  the  haloing  effect  resulting  from  the  infrared  sensor  looking  at  a  body  of  uniform 
temperature.  The  combination  of  image  haloing  and  lack  of  auto-focus  precluded  operators  from 
locating  surface  vessels  when  cued  by  the  ship’s  Combat  Information  Center.  Once  a  target  is 
located  and  auto-focus  is  able  to  engage,  the  sensor  provides  operators  with  a  clear  image. 

Operational  testers  encountered  four  other  payload  anomalies  that  hindered  their  ability 
to  maintain  continuous  target  coverage: 

•  When  operating  in  Location  Mode,  the  payload  would  wander  or  drift  over  time, 
losing  sight  of  the  target. 

•  While  panning,  the  sensor  hits  a  stop  or  limit  obstructing  the  panning  movement, 
causing  operators  to  lose  sight  of  the  target. 

•  The  sensor  would  randomly  slew  off  target  unexpectedly  without  operator  input. 

•  Operators  would  encounter  a  “split  screen”  condition  where  the  top  of  the  sensor 
image  appears  at  the  bottom  of  the  image  on  the  operators  display. 


Previous  Operational  Testing  Results  Revisited  in  IOT&E 

In  support  of  the  RQ-21A  Milestone  C,  the  Navy  conducted  Operational  Test  B-2 
(OT-B2)  from  September  to  November  2012.  Subsequent  to  that  test,  the  Navy  addressed  four 
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of  the  seven  recommendations  contained  in  the  April  2013  DOT&E  Milestone  C  Operational 
Assessment  (updated  the  reliability  growth  curve,  verified  fuselage  structural  integrity,  identified 
and  corrected  the  root  causes  for  the  excessive  number  of  payload  resets,  quantified  and 
compared  the  power  requirements  for  the  system’s  ground  components  to  the  generating 
capabilities  of  standard  Marine  Corps  generators).  The  Navy  did  not  fully  address  two 
recommendations  (strengthening  quality  control  processes  and  communications  relay  payload 
frequency  changes)  that  adversely  affected  system  performance  during  IOT&E.  DOT&E’s  final 
recommendation  to  provide  a  reliability  rationale  in  the  CPD  was  not  addressed  and  would  have 
provided  operational  insight  into  the  poor  system  reliability  demonstrated  during  IOT&E. 

Based  upon  data  collected  during  OT-B2  in  2012,  DOT&E  determined  that  the  electro- 
optical/infrared  payload  demonstrated  the  potential  to  provide  tactical  commanders  with  accurate 
target  acquisition  intelligence.  In  this  report,  DOT&E  determined  that,  while  the  electro- 
optical/infrared  sensor  provides  accurate  target  locations,  operators  using  the  sensors  were 
unable  to  correctly  classify  1  meter  targets  as  required  by  the  CPD,  which  would  allow  for 
consistent  determination  of  hostile/non-hostile  intent  on  the  part  of  individuals  by  identifying 
such  items  as  rifles,  rocket-propelled  grenade  launchers,  and  shovels.  The  difference  between 
DOT&E’s  OT-B2  and  IOT&E  assessment  is  attributable  to  two  differences  between  the  tests: 

•  The  Capability  Development  Document  specified  the  requirement  for  air  vehicle 
flight  at  an  altitude  of  3,000  feet  above  ground  level,  while  the  CPD  specified  flight  at 
an  altitude  of  3,000  feet  above  ground  level  AND  a  sensor  depression  angle  of  45 
degrees,  resulting  in  an  assessment  at  a  slant  range  of  4,242  feet.  Test  results  indicate 
that  probability  of  correct  classification  exceeded  60  percent  at  3,000-foot  slant 
ranges  versus  the  41  percent  at  4,242  foot  slant  ranges. 

•  IOT&E  included  testing  against  maritime  targets  while  OT-B2  did  not  include 
overwater  activities  with  maritime  targets.  During  IOT&E,  RQ-21A  demonstrated 
poor  performance  against  maritime  targets. 

While  the  payload  demonstrated  potential  during  OT-B2,  the  more  stringent  requirements 
and  operating  environment  of  IOT&E  described  above  changed  the  nature  of  the  probability  of 
correct  classification  assessment  between  OT-B2  and  IOT&E  resulting  in  a  less  favorable 
assessment. 

DOT&E  first  identified  the  communications  relay  payload  lack  of  in-flight  frequency 
changes  as  a  potential  deficiency  during  OT-B1  (January  2010)  and  highlighted  this  again  in  the 
2013  OT-B2  memorandum.  In  this  report,  DOT&E  determines  that  the  communications  relay 
payload  limits  the  commanders’  tactical  flexibility. 

System  Performance 
Cybersecurity 

The  adversarial  cybersecurity  assessment  during  the  second  cybersecurity  test  event  on 
USS  Anchorage  found  that  the  Navy  addressed  vulnerabilities  of  the  RQ-21A  electronics 
architecture  discovered  during  the  first  cybersecurity  test  event  at  Twentynine  Palms. 
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Cybersecurity  testing  demonstrated  that  the  system  has  exploitable  vulnerabilities.  The  classified 
appendix  contains  a  complete  description  of  cybersecurity  test  events  and  findings. 
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Section  Four 

Operational  Suitability 


The  RQ-21A  is  not  operationally  suitable.  The  RQ-21A  demonstrated  a  Mean  Flight 
Hour  between  Abort  for  the  System  (MFHBAsys)  of  15.2  hours  versus  the  50-hour  requirement. 
Because  of  air  vehicle  reliability,  overall  system  availability  did  not  meet  the  80  percent  key 
performance  parameter  (KPP)  threshold  (demonstrated  value  =  66.9  percent).  The  average 
service  life  of  the  propulsion  modules  was  48.9  hours,  which  does  not  meet  the  manufacturer’s 
stated  100-hour  capability.  Table  4-1  depicts  system  suitability  parameters,  threshold 
requirements,  and  performance  demonstrated  during  test. 


Table  4-1.  Suitability  Parameters 


Parameter 

Threshold 

Performance 
[80%  confidence  interval] 

Mean  Flight  Hours  between  Abort  -  System  (hours) 

>50 

15.2  [10.6-22.5] 

Mean  Time  between  Aborts  -  Surface  Components  (hours)  Land 

>240 

58.8  [36.2-101.0] 

Mean  Time  between  Aborts  -  Surface  Components  (hours)  Ship 

>240 

36.5  [13.7-137.1] 

Mean  Flight  Hours  between  Abort  -  Air  Vehicle  (hours) 

>60 

35.5  [20.2-67.5] 

Mean  Time  to  Repair  (hours) 

N/A 

1.66  [1.31  -2.10] 

Operational  Availability 

>  80% 

66.9%  [40.1  -96.1] 

Reliability 

Scoring 

The  Navy  defines  Operational  Mission  Failures  (OMFs)  as  a  hardware  failure,  software 
fault,  or  government-furnished  equipment  integration  failure  that  prevented  the  system  from 
performing  one  or  more  mission-essential  functions.  Essential  functions  for  the  RQ-21A  include 
the  following: 

•  Prepare/configure  system  for  use 

•  Collect,  process,  exploit,  and  disseminate  information 

•  Perform  post  mission  tasks 

•  Exchange  information 

An  abort  is  defined  as  a  failure  that  precludes  the  system  from  commencing  or 
completing  a  mission  and  puts  the  system  in  a  not  mission-capable  status. 

The  Navy’s  Commander,  Operational  Test  and  Evaluation  Force  (COMOPTEVFOR) 
collected  failure  data  during  each  phase  of  the  Initial  Operational  Test  and  Evaluation  (IOT&E). 
At  the  conclusion  of  each  event,  scoring  conferences  determined  the  severity  of  each  failure 
incident  using  the  above  definitions.  There  were  scoring  differences  between  the  Navy  and 
DOT&E.  Air  vehicle  failures  in  which  maintainers  replaced  one  air  vehicle  with  another  before 
launch,  thus  delaying  the  start  of  a  mission  for  periods  of  up  to  100  minutes,  were  scored  OMFs 
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by  COMOPTEVFOR  and  not  as  an  abort  even  though  the  system  could  not  “commence  a 
mission”  as  was  prescribed  in  the  definition  of  abort.  DOT&E  scored  this  as  a  system  abort. 

In  general,  OMFs  require  situational  scoring,  in  which  scoring  is  based  on  the 
circumstances  that  exist  at  the  time  of  the  failure.  DOT&E  did  not  score  incidents  as  OMFs. 
Scoring  the  incidents  based  solely  on  the  loss  of  functionality  and  the  duration  of  that  loss 
provides  a  non-situational  method  of  scoring.  This  method  is  independent  of  the  circumstances 
that  exist  during  the  period  functionality  is  lost.  Scoring  incidents  using  the  non-situational 
method  results  in  a  more  conservative  (lower)  reliability  estimate  and,  in  some  respects,  does 
penalize  the  RQ-21A  system  by  ignoring  redundancies  inherent  in  the  system. 

This  report  employs  a  non-situational  scoring  methodology.  Incidents  were  scored  as 
failures  if  they  could  not  be  remedied  quickly,  required  parts,  or  could  not  be  deferred.  Some 
failures  were  further  scored  as  aborts  if  they  constituted  a  major  subsystem  reliability  failure 
during  preflight,  caused  a  crash  or  return  to  base,  caused  a  complete  loss  of  video  or 
communications  with  the  supported  ground  unit,  or  if  the  air  vehicle  was  not  recovered  in  a 
mission-capable  status  because  of  a  reliability  failure  during  flight  or  recovery. 

For  the  purposes  of  calculating  reliability  metrics,  time  is  determined  in  terms  of  flight 
hours  and  operating  hours.  For  the  system  and  air  vehicle,  reliability  requirements  specify  the 
use  of  flight  hours  between  aborts.  For  surface  components,  requirements  specify  the  use  of 
operating  hours  between  aborts.  For  this  case,  calculations  use  ground  control  station  operating 
hours. 

Mean  Flight  Hours  between  Aborts 

Table  4-2  presents  reliability  calculations  for  all  phases  of  IOT&E  along  with  the 
80  percent  confidence  intervals.  The  gray  cells  in  Table  4-2  indicate  the  combinations  of  data 
used  to  compute  each  metric.  In  some  aspects,  the  Camp  Fejeune  results  vary  significantly  from 
the  Twentynine  Palms  and  USS  Anchorage  results.  DOT&E  omitted  Camp  Fejeune  data  from 
some  metrics.1  Because  of  differences  in  the  surface  component  configuration  between 
land-based  and  sea-based  systems,  DOT&E  chose  to  report  Mean  Time  between  Aborts  for 
surface  components  (MTBAsurface  component  (sc>)  for  both  configurations  individually  instead  of 
combining  that  data. 

The  RQ-21A  demonstrated  a  Mean  Flight  Hours  between  Aborts  (MFHBAsystem)  of  15.2 
hours  versus  a  requirement  of  50  hours.  For  the  land-based  configuration,  the  demonstrated 
MTBAsurface  component  is  58.8  hours  versus  a  requirement  of  240  hours.  The  sea-based 
configuration  demonstrated  an  MTBASurface  components  of  36.5  hours  versus  the  same  240-hour 
threshold.  The  RQ-21A  demonstrated  an  MFHBAau  vehicle  of  35.5  hours  versus  the  requirement 
of  50  hours. 


1  Gray-Lewis  test  between  Lejeune  and  Twentynine  Palms  data  demonstrates  a  p-value  of  0.02  for  MFHBASystem 
and  0.053  for  MFFlBAAir  vehicle*  indicating  that  failure  rates  differed  significantly  across  the  two  periods,  and  that 
analysts  should  not  combine  the  data.  The  location  of  the  RQ-21A  launch  and  recovery  site  at  Camp  Lejeune 
was  more  austere  than  the  location  at  Twentynine  Palms,  which  may  have  contributed  to  the  substantially  lower 
MFHBAAir  vehicle  observed  at  Camp  Lejeune. 
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Table  4-2.  Summary  of  Reliability  Results  for  Aborts 


Metric 

(Aborts) 

Test  Event 

Hours 

Aborts 

Performance 
[80%  confidence  interval] 

Threshold 

MFHBAsys 

(hours) 

Twentynine  Palms 

188.3 

12 

15.7  [10.6-24.1] 

>50 

Camp  Lejeune 

20.9 

5 

4.2  [2.3 -8.6] 

USS  Anchorage 

24.4 

2 

12.2  [4.6-45.9] 

Twentynine  Palms  and  USS 
Anchorage  Combined 

212.7 

14 

15.2  [10.6-22.5] 

MTBAsc 

(hours) 

Land 

Twentynine  Palms 

379.6 

6 

63.3  [36.0-  120.4] 

>240 

Camp  Lejeune 

90.6 

2 

45.3  [17.0-170.4] 

Twentynine  Palms  and  Lejeune 
Combined 

470.2 

8 

58.8  [36.2-101.0] 

MTBAsc 
(hours)  Ship 

USS  Anchorage 

72.9 

2 

36.5  [13.7-137.1] 

MFHBAav 

(hours) 

Twentynine  Palms 

188.3 

6 

31.4  [17.9-59.7] 

>60 

Camp  Lejeune 

20.9 

3 

7.0  [3.1  -19.0] 

USS  Anchorage 

24.4 

0 

15.2 

(80%  lower  confidence  bound) 

Twentynine  Palms  and  USS 
Anchorage  Combined 

212.7 

6 

35.5  [20.2-67.5] 

The  Capabilities  Production  Document  (CPD)  does  not  provide  a  rationale  for  the 
50-hour  threshold  requirement.  DOT&E  chose  to  assess  reliability  against  the  probability  of 
completing  a  12-hour  continuous  coverage  mission  without  experiencing  an  in-flight  aborting 
failure  that  causes  the  air  vehicle  to  return  to  base  prematurely.  Once  launched,  the  RQ-21A 
demonstrated  a  mean  time  between  in-flight  mission  aborting  failures  of  46.7  hours  (5  in-flight 
aborting  failures  over  233.6  flight  hours).  This  translates  to  a  75  percent  probability  of 
completing  a  12-hour  on-station  period  without  an  in-flight  aborting  failure  (80  percent 
confidence  interval:  59-87).2 

Two  additional  failures  scored  as  aborts  did  not  result  in  mission  aborts.  During  three 
missions  at  Camp  Lejeune,  the  communications  relay  package  was  not  operational.  The 
RQ-21A  operators  could  not  speak  to  the  supported  ground  unit.  Although  the  communications 
relay  package  failure  occurred  over  multiple  flights,  DOT&E  scored  it  as  a  single  failure  since 
the  payload  never  functioned  properly.  The  second  failure  occurred  on  the  USS  Anchorage. 
Midway  through  the  last  flight,  the  portside  Antenna  Interface  Module  ceased  functioning, 
resulting  in  loss  of  video  and  air  vehicle  command  and  control  on  that  side  of  the  ship. 


2  Assumes  two  air  vehicle  flights,  each  with  six  hours  on-station  time  and  0.62  hours  of  transit  time  for  a  total  of 
6.62-hour  flights  for  each  air  vehicle. 
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Figure  4-1  shows  the  breakdown  of  aborting  failures.  Eight  of  the  19  failures  occurred 
with  the  air  vehicle  in  flight,  resulting  in  mission  aborts  as  follows: 

•  Because  of  a  mission  computer  reset  during  launch,  one  air  vehicle  was  destroyed 
upon  recovery. 

•  Four  failures  required  the  air  vehicle  to  return  to  base  and  abort  the  mission.  These 
failures  included  a  tachometer  generator  failure  alarm,  propulsion  module  unit 
cylinder  head  temperature  discrepancy,  and  two  cases  of  lost  link. 

•  An  air  vehicle  modem  and  an  operator  workstation  networking  issue  caused  the  two 
lost  link  events. 

•  While  the  air  vehicle  was  attempting  to  land,  the  Differential  Global  Positioning 
System  failed,  delaying  recovery  by  97  minutes.  Had  this  failure  occurred  during  a 
longer  flight,  when  the  air  vehicle  was  returning  with  less  fuel,  this  could  have  led  to 
a  crash. 


Ten  preflight  failures  resulted  in  aborts.  Five  of  these  failures  caused  the  flight  aborts. 
The  remaining  five  were  failures  of  a  major  subcomponent  and  caused  the  sortie  to  be  delayed. 
The  major  failures  for  these  events  included: 

•  Air  vehicle  modem 

•  Propulsion  module  unit 

•  GPS  Electronic  Module  software  and  hardware 

•  Air  vehicle  engine  preheater  cable 
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•  Antenna  interface  module 

•  Air  vehicle  avionics  module. 

In  addition,  two  quality  control  problems  caused  aborting  failures: 

•  Ground  control  station  video  settings  incorrectly  configured  at  the  factory 

•  A  CAT-5  cable  on  the  ground  control  station  stack  plugged  into  the  wrong  port  at  the 
factory. 

The  final  abort  was  due  to  the  lack  of  Digital  Terrain  Elevation  Data  maps  on  the 
operator  workstation  to  operate  in  a  maritime  environment.  The  manuals  did  not  provide 
procedures  to  input  elevation  of  sea  level  that  is  necessary  for  maritime  operations. 

Mean  Time  between  Failures 

Although  there  were  no  Navy  reliability  requirements  for  failures,  evaluation  of  the 
failures  provides  insight  into  the  future  maintenance  and  logistics  burden  to  the  Navy  for  the 
system.  Table  4-3  provides  a  summary  of  the  failure  metrics  for  each  phase  of  IOT&E.  The 
gray  cells  in  Table  4-3  indicate  the  combinations  of  data  used  to  compute  each  metric. 

The  RQ-21A  system  demonstrated  a  Mean  Flight  Hours  between  Failure  (MFHBFSys)  of 
4.6  hours.  The  Mean  Time  between  Failures  (MTBF)  for  the  surface  components  in  the  land 
configuration  is  31.3  hours  and  14.6  hours  in  the  ship  configuration.  The  air  vehicle 
demonstrated  a  MTBF  of  5.9  hours. 


Table  4-3.  Summary  of  Reliability  Results  for  Failures 


Metric 

Test  Event 

Hours 

Failures 

Performance 
(80%  confidence  interval) 

MFHBFsystem 

Twentynine  Palms 

188.3 

46 

4.1  [3.4 -5.0] 

Lejeune 

20.9 

14 

1.5  [1.0 -2.2] 

USS  Anchorage 

24.4 

8 

3.4  [2.9 -4.8] 

Twentynine  Palms  and  USS 
Anchorage  Combined 

212.7 

54 

4.6  [3.8 -5.7] 

MTBFsurface  Components 

Land 

Twentynine  Palms 

379.6 

13 

29.2  [20.0-43.9] 

Lejeune 

90.6 

2 

45.3  [17.0 -170.4] 

Twentynine  Palms  and  Lejeune 
Combined 

470.2 

15 

31.3  [22.1  -45.7] 

MTBFsurface  Components 

Ship 

USS  Anchorage 

72.9 

5 

14.6  [7.9-30.0] 

M  F  FI  B  FAir  Vehicle 

Twentynine  Palms 

188.3 

33 

5.7  [4.5 -7.3] 

Lejeune 

20.9 

12 

8.1  [3.7-22.1] 

USS  Anchorage 

24.4 

3 

4.8  [4.0 -5.9] 

Twentynine  Palms  and  USS 
Anchorage 

212.7 

36 

5.9  [4.7  -  7.4] 
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Figure  4-2  presents  a  breakdown  of  failure  chargeability.  There  were  68  failures  during 
the  IOT&E  (this  includes  Camp  Lejeune  failures).  Air  vehicle  components  accounted  for 
68  percent  (46  of  68)  of  the  failures.  The  surface  components  contributed  29  percent  (20  of  68) 
of  the  failures.  Inaccurate/incomplete  technical  publications  caused  the  remaining  3  percent 
(2  of  68)  of  the  failures.  This  figure  shows  the  severity  of  the  failures;  in  the  figure  key,  the  dark 
blue,  green,  and  red  colors  represent  the  aborts  and  the  light  blue,  green  and  red  colors  represent 
the  failures.  It  breaks  down  the  failure  mode  by  cause:  hardware,  software,  or  other  (technical 
publications  or  built-in  test). 

The  four  most  common  failure  modes  accounted  for  54  percent  (37  of  68)  of  all  failures. 

•  Structural  damage  to  the  air  vehicle  accounted  for  23  percent  (16  of  68).  The  forces 
experienced  by  the  air  vehicle  during  capture  with  the  recovery  system  cause  parts  to 
break.  This  damage  ranged  from  hinge  clips  to  replacement  of  entire  wing  and 
empennage  modules. 

•  Payload  software  accounted  for  15  percent  (10  of  68)  of  the  failures.  Operators 
remedied  these  failures,  such  as  pixilated  imagery,  loss  of  payload  control,  and  the 
payload  slewing  away  automatically,  by  resetting  payload  software. 

•  Nine  percent  (6  of  68)  of  failures  were  operator  workstation  software. 

•  Propulsion  module  units  contributed  7  percent  (5  of  68)  of  all  failures. 

Air  Vehicle  Heater  Cable 
Built-In  Test 
Frangible  Bolts  (Packaging) 

Communications  Relay  Payload 
Technical  Publications 
GPS  Electronics  Module 
Ground  Data  Terminal 
Antenna  Interface  Module 
Avionics  Module 
Propulsion  Module  Unit 
Modem 
Operator  Work  Station 
Payload 
Air  Vehicle  (Structural) 


Number  of  Incidents 


Figure  4-2.  Failure  chargeability 
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■  Other  -  abort 

■  Other  -  failure 


10 


12 


14 


16 


28 


Propulsion  Module  Units 


Propulsion  module  units  have  a  manufacturer-stated  service  life  of  100  operating  hours. 
Upon  reaching  100  hours,  maintainers  replace  the  propulsion  module  unit  and  return  the 
high-time  unit  to  the  manufacturer  for  refurbishment.  Over  the  course  of  995  flight  hours,  the 
Early  Operational  Capability  detachment  replaced  26  propulsion  modules.  Figure  4-3  presents 
the  reasons  for  propulsion  module  replacements  and  the  distributions  of  module  service  hours  at 
the  time  of  replacement  for  each  of  the  26  modules.  This  resulted  in  a  median  life  span  for  the 
propulsion  modules  of  42.2  hours  (average  life  span  48.9  hours). 


Replacement 

Reason 

Count 

Fuel  Leak 

9 

Low  RPMs 

6 

Pulley  Cracks 

4 

Reached  End  of  Life1 

3 

Cylinder  Head  Temp 

3 

Fuel  Pump 

1 

Note:  Two  propulsion  module  units  reached  100-hour 
life  limit,  one  reached  80-hour  adjusted  life  limit. 
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Figure  4-3.  Reasons  for  Propulsion  Module  Replacement  and  Distribution  of  Lifespans 


On  Day  103  of  the  Early  Operational  Capability  deployment,  after  two  previous  in-flight 
engine  failures,  and  the  discovery  of  cracks  on  the  propulsion  module  pulley  cranks  on  other  air 
vehicles,  the  detachment  lowered  service  life  to  80  hours.  On  day  1 13  of  the  deployment,  the 
detachment  further  reduced  expected  propulsion  module  service  life  to  50  hours  to  be  more 
aligned  with  the  demonstrated  average  life  span  of  the  propulsion  modules. 


Availability 

The  RQ-21A  demonstrated  an  availability  rate  of  66.9  percent  (148.5  down  hours  during 
449  test  hours)  during  operations  at  Twentynine  Palms.  The  demonstrated  availability  does  not 
meet  the  80  percent  Operational  Availability  KPP. 

The  CPD  describes  an  available  system  as  follows: 

•  2  of  5  air  vehicles  mission-capable 

•  1  of  2  Ground  Control  stations  mission-capable  (each  ground  control  station  consists 
of  two  operator  workstations) 

•  A  mission-capable  launcher 

•  A  mission-capable  Small  Tactical  Unmanned  Aircraft  System  (STUAS)  Recovery 
System. 
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Figure  4-4  presents  the  status  of  each  RQ-21A  component  during  operations  at 
Twentynine  Palms.  The  period  between  the  January  15  air  vehicle  mishap  until  the  morning  of 
January  26,  2014  is  not  counted  in  the  availability  calculation.  During  this  period,  the  Marine 
Corps  suspended  flight  operations  pending  the  outcome  of  the  mishap  investigation. 
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Figure  4-4.  RQ-21A  System  Availability  at  Twentynine  Palms 

A  lack  of  air  vehicles  caused  148  of  the  148.5  total  system  down  hours.  During  those 
148  hours,  the  system  did  not  have  the  minimum  two  mission-capable  air  vehicles.  The  STUAS 
Recovery  System  accounted  for  the  remaining  0.5  hours  of  system  down  time.  Operator 
workstations  and  the  launcher  did  not  account  for  any  system  down  time. 

The  loss  of  Air  Vehicle  1010  on  the  first  IOT&E  flight  without  replacement  adversely 
affected  system  availability.  Instead  of  five  air  vehicles  being  available  to  make  the  two 
mission-capable  air  vehicles  system  requirement,  the  mishap  left  the  system  with  four  air 
vehicles  to  make  two  mission-capable  air  vehicles  system  requirement,  thereby  reducing  built-in 
system  redundancy.  To  assess  the  impact  of  the  loss  of  Air  Vehicle  1010  without  replacement, 
DOT&E  notionally  included  a  fifth  aircraft  as  if  it  were  available  for  operations  and  conducted  a 
bootstrap  analysis  to  compare  system  availability  with  four  and  five  air  vehicles  to  account  for 
the  loss  of  air  vehicle  1010  for  the  duration  of  operations  at  Twentynine  Palms.  Bootstrapping  is 
a  resampling  method  that  relies  on  random  sampling  with  replacement.  Bootstrapping  allows 
assigning  measures  of  accuracy  (in  this  case  confidence  intervals)  to  sample  estimates.  10,001 
bootstrap  data  sets  were  generated  by  re-sampling  the  original  data  set  of  system  up  and  down 
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times  with  replacement,  generating  a  bootstrap  distribution  for  System  Availability.  DOT&E 
derived  confidence  intervals  based  on  the  10th  and  90th  percentiles  of  this  availability  bootstrap 
distribution.  Figure  4-5  presents  the  results  of  this  bootstrap  analysis.  There  is  more  variability 
in  the  results  with  four  air  vehicles  as  shown  by  the  dispersion  of  the  distribution  (length  of 
vertical  lines).  Conversely,  the  bootstrapping  results  for  five  air  vehicles  demonstrates  that  there 
was  a  higher  availability  and  lower  variability  (dispersion)  as  seen  in  the  shorter  vertical  line 
length.  The  box  plot  for  5 -Air  Vehicles  does  include  some  outliers,  symbolized  as  dots  at  the 
bottom  of  the  figure,  but  the  number  is  small  and  falls  outside  the  line. 


Figure  4-5.  Distribution  of  Bootstrap  Availability  Results  by  Number  of  Air  Vehicles 


Using  the  bootstrap  methodology,  system  availability  exceeded  80  percent  for  39.6 
percent  of  the  four-air  vehicle  system  iterations.  For  the  five-air  vehicle  system,  61  percent  of 
the  bootstrap  iterations  exceeded  the  80  percent  availability  threshold.  While  the  presence  of  a 
fifth  air  vehicle  might  have  allowed  the  system  to  demonstrate  meeting  the  80  percent 
availability  threshold,  it  would  not  have  done  so  with  statistical  confidence. 

Air  Vehicle  Availability 

As  described  above,  air  vehicles  became  the  driving  factor  for  system  availability. 
Figure  4-6  presents  the  number  of  mission-capable  air  vehicles  over  the  course  of  testing  at 
Twentynine  Palms.  The  red  region  in  the  left  side  of  Figure  4-6  (depicting  either  one  or  no  air 
vehicles)  represents  periods  when  the  overall  system  is  in  a  down  status  because  of  air  vehicle 
availability.  As  shown  in  right  side  of  Figure  4-6,  individual  air  vehicle  availability  averaged 
51.4  percent.  The  IOT&E  test  unit  required  an  influx  of  spare  parts  on  February  6,  2014  to 
complete  the  Twentynine  Palms  phase  of  testing.  Without  the  influx  of  spare  parts,  overall 
system  availability  would  have  been  significantly  less  than  the  demonstrated  66.9  percent. 


3  The  length  of  the  line  is  equal  to  1 .5  times  the  Interquartile  Range. 
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Figure  4-6.  Number  of  Mission-Capable  Air  Vehicles  over  Time  and  Air  Vehicle  Availability 


Air  vehicle  availability  was  poor  for  the  Early  Operational  Capability  detachment.  This 
detachment  started  operations  with  seven  air  vehicles,  with  an  eighth  air  vehicle  delivered  on 
Day  57  of  the  deployment.  As  shown  in  Figure  4-7,  after  119  days,  the  Early  Operational 
Capability  detachment  had  two  mission-capable  air  vehicles  (25  percent).  Four  of  eight  air 
vehicles  were  beyond  repair  (one  cracked  fuselage,  two  in-flight  engine  failures,  and  one 
damaged  fuselage  power  distribution  board)  and  two  air  vehicles  required  propulsion  modules. 
Of  the  two  operable  air  vehicles,  one  had  8  hours  remaining  on  its  propulsion  module  unit  and 
would  then  require  a  replacement,  which  the  detachment  did  not  possess. 


1  -  Arrival  of  8"1  air 
vehicle 

2 -Air  Vahids 
rendered  unusable 
a  -  Cracked  fuselage 
b  -  Engine  failure 
c-  Engine  Failure 
d  -  Avia  nice  board 

3  -  Propulsion 
Module  service  life 
decrease 

a  - 1 QQ  hours  to  60 
hours 

b-8Q  hours  to  60 
hours 


Figure  4-7.  Early  Operational  Capability  Air  Vehicle  Availability 


Maintainability 

Operators,  maintainers,  and  field  service  representatives  performed  three  types  of 
maintenance:  pre-  and  post-flight,  scheduled,  and  unscheduled  maintenance.  Pre-  and 
post-flight  maintenance  includes  aircraft  preparation  and  recovery.  Scheduled  maintenance 
includes  events  delineated  in  maintenance  manuals  to  be  performed  at  a  specific  time,  e.g., 
25-hour  engine  inspection.  Failures  that  were  quickly  remedied,  did  not  require  parts,  or  could 
be  deferred  are  referred  to  as  other  maintenance  actions.  These  are  also  included  in  the 
maintenance  burden  calculations. 
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During  the  testing  at  Twentynine  Palms,  maintainers  expended  65.1  hours  repairing 
44  failures.4, 5  The  distribution  of  repair  times  is  shown  in  Figure  4-8.  Including  the  other 
maintenance  actions,  the  total  unscheduled  maintenance  time  was  77.7  hours.  Maintainers  and 
field  service  representatives  spent  an  additional  246.9  hours  conducting  scheduled  and  pre-  and 
post-flight  maintenance.  The  total  maintenance  time  expended  was  324.6  hours.  The 
maintenance  man  hour  per  flight  hour  is  1.7  hours  (324.6  hours/188.3  hours).  The  unscheduled 
maintenance  man  hour  per  flight  hour  is  0.41  (77.7/188.3).  Assuming  a  lognormal  distribution, 
the  mean  time  to  repair  is  1.66  hours  [1.31-2.10],  The  mean  time  to  repair  aborts  is  3.92  hours 
[2.57-5.98], 
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Figure  4-8.  Distribution  Twentynine  Palms  Repair  Times 


Logistics  Supportability 
Technical  Publications 

Unlike  manned  aircraft,  the  RQ-21A  does  not  have  a  “pocket  checklist”  for  operator  use 
during  start-up  and  recovery,  and  when  reacting  to  emergencies.  Instead,  operators  must  balance 
a  4-inch,  914-page  binder  on  their  laps  while  conducting  normal  and  emergency  procedures. 

This  cumbersome  and  time-consuming  process  diverts  operator  attention  from  the  problem  at 
hand. 

Organizational  level  maintenance  procedures  stop  at  the  module  level.  During  shipboard 
operations,  Air  Vehicle  1002  reported  a  bad  propulsion  module  generator  during  start-up. 
Maintainers  installed  a  new  propulsion  module  on  Air  Vehicle  1002,  which  indicated  a  bad 
generator  on  the  new  propulsion  module.  At  that  point,  maintainers  suspected  that  Air  Vehicle 
1 002  had  an  electrical  problem  and  that  both  propulsion  module  generators  were  in  fact 
operating  normally.  Because  maintenance  manuals  did  not  contain  troubleshooting  procedures 
for  the  propulsion  modules,  the  option  available  to  the  Marines  was  to  pack  up  both  propulsion 


4  Maintainability  data  from  Twentynine  Palms  were  the  only  data  adequate  for  analysis. 

5  There  are  two  fewer  failures  here  than  were  accounted  for  in  Table  4-3.  These  failures  were  not  collected  by 
COMOPTEVFOR,  but  were  observed  by  DOT&E  during  the  test. 
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modules  and  return  them  to  the  manufacturer.  The  incorporation  of  module  troubleshooting 
procedures  would  reduce  administrative  logistics  delay  time  and  the  number  of  spare  parts  in  the 
pack-up  kit. 

To  support  shipboard  operations,  the  Navy  permanently  installed  some  RQ-21 A  ground 
components  (antennae  interface  modules,  data  link  antennae)  on  selected  ships.  The  ships’ 
personnel,  and  not  the  RQ-21  A  detachment,  own  and  maintain  these  components.  None  of  the 
ships’  personnel  received  training  on  maintaining  these  installed  components.  The  ship  did  not 
receive  spare  parts,  maintenance  manuals,  or  wiring  diagrams  with  which  to  facilitate  repairs. 
During  the  last  IOT&E  flight,  the  port-side  antennae  interface  module  failed.  As  a  result,  the 
operators  could  not  fly  the  air  vehicle  on  the  port  side  of  the  ship.  Without  trained  personnel  and 
spare  parts,  this  failure  could  have  precluded  further  tactical  flight  operations. 

After  the  Marine  Corps  declares  the  fielding  of  an  RQ-21  A  Initial  Operational  Capability, 
they  plan  to  rely  upon  field  service  representatives  to  perform  organizational  level  maintenance 
for  two  years.  After  the  initial  two  years,  the  Marine  Corps  will  decide  whether  to  train  Marine 
maintainers  or  continue  to  rely  upon  field  service  representatives. 

During  this  two-year  period,  field  service  representatives  are  required  to  conduct 
maintenance  in  accordance  with  the  Naval  Aviation  Maintenance  Program  (Commander  Naval 
Air  Forces  Instruction  4790. 2A).  Instead  of  using  the  established  Naval  Aviation  Logistics 
Command  Management  Information  System  to  track  maintenance  and  flight  information,  the 
field  service  representatives  use  an  Insitu  program  called  Sapphire. 

At  the  completion  of  testing,  operational  testers  conducted  a  comparison  of  paper 
maintenance  action  forms  completed  by  Marines  to  the  Sapphire  database.  Of  the  136 
unscheduled  maintenance  actions  completed  over  the  course  of  IOT&E,  Sapphire  accurately 
archived  15. 

Because  Sapphire  was  not  available  during  IOT&E  operations  at  Twentynine  Palms  and 
Camp  Lejeune,  field  service  representatives  attempted  to  enter  data  after  the  completion  of  each 
test  period.  Instead  of  allowing  maintainers  to  retroactively  enter  data  using  actual  times  and 
dates  for  each  maintenance  action,  Sapphire  automatically  enters  the  current  date  and  time  as  the 
failure  time.  Given  the  delays  between  the  test  periods  and  when  data  were  entered  into 
Sapphire,  this  created  a  30-day  lag  in  maintenance  tracking  data. 

Supply  Support 

The  lack  of  spare  parts  resulted  in  an  administrative  logistics  delay  time  of  27  days. 
Administrative  logistics  delays  fell  into  two  categories:  items  not  received  at  the  start  of  the 
IOT&E  and  lack  of  air  vehicle  spare  parts.  Twelve  items  that  provide  system  redundancy  did  not 
arrive  with  the  system  at  the  start  of  IOT&E,  including  a  single  refueler  when  a  unit  should  have 
two,  and  a  single  pitot  system  test  set  when  the  unit  should  have  two.  The  test  unit  did  not 
receive  these  12  items  for  34  days.  The  lack  of  the  second  refueler  caused  mission  aborts  and 
required  the  test  unit  to  borrow  equipment  from  the  Early  Operational  Capability  detachment 
(which  had  yet  to  deploy)  to  commence  flight  operations. 
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Lack  of  air  vehicle  spare  parts  caused  administrative  logistic  delays.  In  six  instances,  air 
vehicles  spent  a  significant  amount  of  time  in  a  non-mission  capable  status  while  awaiting  spare 
parts  (7,  9,  13,  15,  15,  and  19  days).  Propulsion  modules  caused  four  of  the  extended  delays,  an 
avionics  module  caused  one,  while  an  empennage  module  caused  the  last.  Extended  logistics 
delays  increased  system  down  time,  and  without  the  use  of  unacceptable  workarounds,  would 
have  halted  flights  during  operational  testing. 

Production  quality  control  issues  contributed  to  the  system’s  poor  reliability  and 
availability.  In  seven  instances,  operators  and  maintainers  found  incorrectly 
assembled/configured  components  during  initial  system  set-up,  maintenance  inspections,  and 
pre-flight  (see  Table  4-4).  Incorrectly  assembled/configured  components  caused  increased 
maintenance  time  to  replace  components,  resulting  in  reduced  availability. 


Table  4-4.  Production  Quality  Issues 


Component 

Discrepancy 

Air  Vehicle  Fueler 

Air  vehicle  fueler  would  not  prime 

Air  Vehicle 

New  Modems  were  not  properly  configured 

Broken  servo  connector  and  retention  clip 

Avionics  module  software  not  loaded 

Lost  link 

Operator  Workstation 

Network  communications  issue 

Operator  Workstation  not  displaying  video 

Support  Equipment 

RQ-21A  support  equipment  does  not  facilitate  expeditionary  operations.  As  fielded,  each 
RQ-21A  system  includes  two  tents  used  as  shelter  for  the  ground  control  stations.  The  system 
does  not  provide  for  inclement  weather  protection  for  air  vehicles,  support  equipment,  and  spare 
parts.  As  shown  in  Figure  4-9,  air  vehicle  module  packing  cases  and  spare  parts  would  be  stored 
under  a  tarpaulin  strung  between  two  tents  or  in  the  back  of  High  Mobility  Multipurpose 
Wheeled  Vehicles  (HMMWVs). 


Figure  4-9.  RQ-21A  Expeditionary  Configuration 
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Three  module  containers  for  each  air  vehicle  are  supposed  to  provide  weather  protection 
between  flights: 

•  One  for  the  nose  and  electro-optical/infrared  sensor 

•  One  for  the  pair  of  wings 

•  One  for  the  fuselage,  propulsion  module,  and  empennage 

The  RQ-21 A  unit  operated  from  a  permanent  hangar  while  operating  at  Twentynine 
Palms.  During  that  month-long  period,  the  unit  did  not  notice  any  container-related  issues. 
During  1  ldays  of  operations  at  Camp  Lejeune,  where  containers  and  equipment  were  stored  out 
in  the  elements  (as  depicted  in  Figure  4-9),  maintainers  discovered  six  instances  of  storage 
container  failure: 

•  On  two  occasions,  foam  cushion  inserts  separated  from  the  wing  container.  This 
allowed  two  wings  to  come  in  contact  and  break  frangible  winglet  bolts 

•  The  design  of  the  fuselage  container  allows  stacking  two  high.  On  one  occasion, 
stacking  containers  caused  a  6-  inch  hole  in  the  bottom  container. 

•  After  overnight  rain,  maintainers  discovered  approximately  16  ounces  of  water  on  the 
bottom  of  two  wing  containers  and  one  fuselage  container. 

Ship  Compatibility 

Because  of  the  ambient  sea-states  during  the  developmental  test  ship  periods, 
developmental  testing  was  not  able  to  certify  RQ-21  A  launch  and  recovery  operations  to  the  full 
envelope  specified  by  the  KPP  (see  Table  4-5). 


Table  4-5.  Launch  and  Recovery  Pitch  and  Roll 

Limits 

Ships  Pitch 

Ships  Roll 

Required  Values  for  Launch  and  Recovery 

-3  to  +3  degrees 

-5  to  +5  degrees 

Approved  Launch  Limits 

-1.1  to  0  degrees 

-1.6  to  +2.8  degrees 

Approved  Recovery  Limits 

-1 .3  to  +0.1  degrees 

-2.6  to  +1.0  degrees 

The  approved  pitch  and  roll  limits  did  not  limit  operations  during  the  IOT&E  ship  period. 
While  the  current  pitch  and  roll  limits  do  not  meet  the  KPP  threshold,  they  will  support  initial 
shipboard  deployments. 

The  shipboard  antenna  system  design  includes  two  directional  antennas  on  the  forward 
superstructure  to  provide  360-degree  data  link  coverage  for  ranges  between  10  and  50  nautical 
miles.  An  omni-directional  antenna  provides  data  link  coverage  inside  of  10  nautical  miles. 
During  IOT&E  ship-based  testing,  operators  discovered  a  blind  spot  in  directional  antenna 
coverage  aft  of  the  ship.  This  blind  spot  restricts  RQ-21  A  operations  beyond  10  nautical  miles 
aft  of  the  ship. 
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T  ransportability 

All  RQ-21A  associated  vehicles  and  trailers  received  certification  for  transport  by 
CH-53E  helicopters  as  required  by  the  air  transportability  KPP.  All  trailers  meet  the  weight  and 
structural  limits  to  allow  for  towing  behind  HMMWVs  as  required  by  the  ground  transportability 
key  performance  parameter. 

The  Marine  Corps  intends  the  ground  transport  configuration  (four  HMMWVs,  two  cargo 
trailers,  STUAS  Recovery  System  trailer,  and  launcher  trailer)  to  provide  a  fully  functional 
RQ-21A  detachment  to  support  expeditionary  operations.  As  currently  configured,  an 
expeditionary  detachment  requires  significant  support  in  order  to  conduct  flight  operations. 

The  expeditionary  detachment  does  not  include  a  Marine  communications  specialist, 
whose  presence  is  required  to  upload  air  vehicle  cryptographic  keys  before  every  flight  and  to 
maintain  radios  used  by  the  ground  control  station  to  communicate  with  other  units.  The 
expeditionary  configuration  lacks  dedicated  generators  to  power  the  ground  control  stations, 
launcher,  and  recovery  system.  The  RQ-21 A  detachment  would  rely  upon  supported  units  to 
provide  generators  and  the  mechanics  to  support  them.  It  is  not  realistic  to  expect  supported 
ground  units  operating  in  an  expeditionary  manner  would  have  an  excess  of  generators  and 
communications  specialists  to  dedicate  to  the  RQ-21  A  detachment. 

Safety 

One  contributing  factor  to  the  Air  Vehicle  1010  mishap  is  that  the  RQ-21  A  Naval  Air 
Training  and  Operations  Standardization  manual  does  not  support  safe  air  vehicle  operations. 

The  manual  does  not  contain  important  information  regarding  mission  computer  logic  during 
normal  operations.  This  lack  of  information  is  exacerbated  during  emergencies  where  operators 
are  not  aware  of  which  conditions  enable/disable  various  aspects  of  air  vehicle  functionality. 

This  lack  of  system  operations  information  directly  resulted  in  the  loss  of  Air  vehicle  1010 
during  the  first  IOT&E  flight. 

On  at  least  three  occasions  during  IOT&E,  disagreements  arose  between  the  RQ-21  A 
operators  and  the  field  service  representatives.  In  each  case,  the  field  service  representative 
provided  guidance  that  contradicted  operator  training  and  the  Naval  Air  Training  Operations  and 
Standardization  manual.  Differences  between  operator  training  and  field  service  representative 
guidance  could  lead  to  confusion  during  critical  phases  of  flight. 

The  air  vehicle  refueling  system  displays  fuel  quantities  in  pounds.  During  the  start-up 
process,  operators  manually  initialize  air  vehicle  fuel  quantity  using  kilograms.  Operators  rely 
upon  non-standardized  handheld  calculators  to  convert  between  pounds  and  kilograms.  This 
could  cause  operators  to  launch  air  vehicles  with  incorrect  fuel  quantities,  resulting  in  possible 
overstress  of  the  aircraft  upon  launch  or  fuel  starvation  during  flight. 

Previous  Operational  Testing  Results  Revisited  in  IOT&E 

Just  prior  to  OT-B2,  developmental  testing  experienced  a  fuselage  failure  that  resulted  in 
a  cracked  airframe,  possibly  attributable  to  production  quality  control  issues.  DOT&E 
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recommended  that  the  Navy  strengthen  the  quality  control  process  in  those  areas  where 
manufacturing  shortcomings  could  result  in  air  vehicle  losses  in  order  to  minimize  the  potential 
for  future  mishaps.  During  IOT&E,  production  quality  control  issues  contributed  to  the  system’s 
poor  reliability  and  availability.  In  seven  instances,  operators  and  maintainers  found  incorrectly 
assembled/configured  components  during  initial  system  setup,  maintenance  inspections,  and 
while  conducting  system  preflight  procedures.  These  instances  increased  maintenance  time  and 
reduced  availability. 

During  OT-B2,  the  system  demonstrated  a  Mean  Flight  Hours  between  Abort  of 
10.6  hours,  well  below  the  threshold  requirement  of  50  flight  hours.  The  demonstrated  Mean 
Flight  Hours  between  Abort  of  15.2  hours  during  IOT&E  continues  to  be  well  below  the 
threshold  requirement.  This  contributed  towards  DOT&E  finding  the  RQ-21A  not  suitable. 
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Section  Five 

Recommendations 

The  Navy  and  Marine  Corps  should  consider  the  following  recommendations  in  order  to 
improve  operational  effectiveness  and  operational  suitability  before  entering  full-rate  production 
and  should  verify  the  corrections  to  deficiencies  during  follow-on  test  and  evaluation. 

Operational  Effectiveness 

•  Improve  sensor  resolution  to  increase  probability  of  correct  classification  for  1  -meter 
sized  targets. 

•  Increase  the  number  of  programmed  air  vehicle  loiter  patterns  available  to  increase 
tactical  flexibility  and  reduce  fuselage  obstruction  of  the  payload. 

•  Fully  integrate  the  communications  relay  payload  into  the  air  vehicle  architecture  to 
allow  for  in-flight  frequency  changes  and  altering  of  other  radio  settings  to  increase 
tactical  utility. 

•  Update  the  Sensor  Resolution  and  Night  Sensor  Resolution  key  performance 
parameter  probability  of  correct  classification  threshold  values  of  50  percent  to  a 
more  tactically  meaningful  value. 

•  Implement  the  cybersecurity  recommendations  in  the  classified  annex  to  improve 
system  security. 

•  Address  payload  anomalies  to  reduce  the  frequency  and  duration  of  periods  where 
operators  lose  positive  identification  of  a  target. 

•  Improve  infrared  sensor  performance  in  a  maritime  environment  to  eliminate  haloing. 

•  Review  the  communications  relay  payload  architecture  to  allow  support  of  frequency 
hopping  radios. 

•  Conduct  additional  infrared  marker  operational  testing  with  air  and  ground  observers 
to  fully  characterize  performance. 

•  Increase  the  power  of  the  laser  marker  to  increase  its  utility  to  ground  forces. 

•  Simplify  communications  relay  payload  programming  to  reduce  workload  and  limit 
potential  errors. 

•  Determine  the  cause  of  the  systematic  error  in  the  electro-optical/infrared  payload  to 
increase  sensor  point  of  interest  accuracy. 

•  Incorporate  laser  rangefinder  information  into  the  target  location  calculations  to 
increase  sensor  point  of  interest  accuracy. 

•  Conduct  additional  operational  testing  of  the  communications  relay  payload  secure 
data  capability  to  fully  characterize  performance. 
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•  Restructure  the  personnel  and  equipment  in  the  expeditionary  RQ-21A  detachment  to 
be  fully  self-reliant  and  reduce  their  burden  to  supported  units. 

Operational  Suitability 

•  Increase  propulsion  module  service  life  and  reliability  to  reduce  maintainer  workload, 
the  number  of  spares,  and  operating  costs. 

•  Expand  the  systems  description  and  flight  characteristics  section  of  the  RQ-21A 
Naval  Air  Training  and  Standardization  Procedures  manual  to  allow  operators  to 
safely  react  to  system  emergencies. 

•  Increase  production  quality  control  and  implement  thorough  acceptance  procedures 
for  delivered  systems  and  spare  parts  to  reduce  the  number  of  faulty  items  received  by 
Fleet  operators. 

•  Fully  train  and  provide  ship  personnel  with  technical  manuals,  wiring  diagrams,  and 
spare  parts  related  to  RQ-21A  shipboard  components  to  increase  RQ-21A  full 
mission  capability. 

•  Require  field  service  representatives  to  utilize  the  Naval  Aviation  Logistics 
Command/Management  Information  System  to  track  system  maintenance  to  provide 
the  Marine  Corps  with  better  maintenance  data  regarding  the  maintenance  burden 
associated  with  RQ-21A  operations. 

•  Undertake  a  comprehensive  review  of  failure  modes  to  improve  system  reliability  and 
reduce  spare  parts. 

•  Increase  expenditures  on  spare  parts  to  reduce  administrative  logistics  delay  times. 

•  Increase  the  number  of  spare  parts  in  the  pack-up  kit  to  increase  system  availability. 

•  Adjust  coverage  of  the  shipboard  directional  antenna  system  to  provide  full 
360-degree  coverage. 

•  Produce  a  pocket  checklist  from  the  RQ-21 A  Naval  Air  Training  and  Standardization 
Procedures  manual  to  allow  operators  to  expeditiously  complete  required  procedural 
steps/checks  and  respond  to  emergencies. 

•  Include  communications  relay  payload  operations,  programming,  and  repairs  in  the 
field  service  representative,  maintainer,  and  operator  syllabi  to  reduce  crew  workload 
and  limit  potential  errors. 

•  Review  air  vehicle  module  maintenance  procedures  at  the  organizational  level  to 
reduce  administrative  logistics  delay  times  and  the  number  of  required  spare  parts. 

•  Provide  each  RQ-21  A  detachment  with  a  maintenance  tent  to  protect  spare  parts  and 
air  vehicle  modules  during  inclement  weather. 

•  Conduct  additional  developmental  testing  to  expand  the  port-side  recovery  wind 
envelope  to  increase  shipboard  flight  operations  flexibility. 
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•  Conduct  additional  developmental  testing  to  increase  ship  pitch  and  roll  limits  for 
RQ-21A  launch  and  recovery. 

•  Improve  air  vehicle  fuselage  and  wing  module  containers  to  protect  against  weather 
and  prevent  component  damage  from  the  failure  of  internal  packing  and  support 
structures. 

•  Standardize  the  unit  of  measure  for  fuel-related  quantities  throughout  the  RQ-21A 
system. 
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